Selective update of inference models

ABSTRACT

Systems and methods for selective update of inference models are provided. For example, a first inference model and a second inference model may be obtained, and the first inference model and the second inference model are based on a first version of a set of training examples. Further, in some examples, an updated version of the set of training examples may be obtained. Further, in some examples, an updated version of the first inference model may be generated based on the updated version of the set of training examples. Further, in some examples, the updated version of the first inference model may be evaluated, and the evaluation result may be used to determine whether to update the second inference model.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/685,914, filed on Jun. 16, 2018. The entirecontent of all of the above-identified application is hereinincorporated by reference.

BACKGROUND Technological Field

The disclosed embodiments generally relate to systems and methods forselective update of inference models. More particularly, the disclosedembodiments relate to systems and methods for selective update ofinference models based on results of updates to other inference models.

Background Information

Computerized devices are now prevalent, and data produced and maintainedby those devices is increasing.

Audio sensors are now part of numerous devices, and the availability ofaudio data produced by those devices is increasing.

Image sensors are now part of numerous devices, from security systems tomobile phones, and the availability of images and videos produced bythose devices is increasing.

Machine learning algorithms, that use data to generate insights, rulesand algorithms, are widely used.

SUMMARY

In some embodiments, systems and methods for the creation, maintenance,and usage of datasets and annotations are provided.

In some embodiments, systems and methods for selective update ofinference models are provided.

In some embodiments, a first inference model and a second inferencemodel may be obtained, and the first inference model and the secondinference model are based on a first version of a set of trainingexamples. Further, in some examples, an updated version of the set oftraining examples may be obtained. Further, in some examples, an updatedversion of the first inference model may be generated based on theupdated version of the set of training examples. Further, in someexamples, the updated version of the first inference model may beevaluated, and the evaluation result may be used to determine whether toupdate the second inference model.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store data and/or computerimplementable instructions for carrying out any of the methods describedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are block diagrams illustrating some possibleimplementations of a communicating system.

FIGS. 2A and 2B are block diagrams illustrating some possibleimplementations of an apparatus.

FIG. 3 is a block diagram illustrating a possible implementation of aserver.

FIGS. 4A and 4B are block diagrams illustrating some possibleimplementations of a cloud platform.

FIG. 5 is a block diagram illustrating a possible implementation of acomputational node.

FIG. 6 illustrates an exemplary embodiment of a memory storing aplurality of modules.

FIG. 7 illustrates an example of a method for presenting inferencemodels based on interrelationships among inference models.

FIG. 8 is a schematic illustration of an example output consistent withan embodiment of the present disclosure.

FIG. 9 illustrates an example of a method for personalized selection ofinference models based on usage of other inference models.

FIGS. 10A, 10B, 100, 10D, 10E, 10F and 10G are schematic illustrationsof different configurations for using inference models consistent withembodiments of the present disclosure.

FIG. 11 illustrates an example of a method for selective update ofinference models.

FIG. 12 is a schematic illustration of example an environment consistentwith an embodiment of the present disclosure.

FIG. 13 illustrates an example of a method for selecting hyperparameters for machine learning algorithms based on past trainingresults.

FIG. 14A is a schematic illustration of a user interface consistent withan embodiment of the present disclosure.

FIG. 14B is a schematic illustration of a user interface consistent withan embodiment of the present disclosure.

FIG. 15 illustrates an example of a method for class aware objectmarking.

FIGS. 16A, 16B, 16C, 16D, 16E and 16F are schematic illustrations ofimages consistent with embodiments of the present disclosure.

FIG. 17 illustrates an example of a method for selecting images formanual annotation.

DESCRIPTION

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing”, “calculating”,“computing”, “determining”, “generating”, “setting”, “configuring”,“selecting”, “defining”, “applying”, “obtaining”, “monitoring”,“providing”, “identifying”, “segmenting”, “classifying”, “analyzing”,“associating”, “extracting”, “storing”, “receiving”, “transmitting”, orthe like, include action and/or processes of a computer that manipulateand/or transform data into other data, said data represented as physicalquantities, for example such as electronic quantities, and/or said datarepresenting the physical objects. The terms “computer”, “processor”,“controller”, “processing unit”, “computing unit”, and “processingmodule” should be expansively construed to cover any kind of electronicdevice, component or unit with data processing capabilities, including,by way of non-limiting example, a personal computer, a wearablecomputer, a tablet, a smartphone, a server, a computing system, a cloudcomputing platform, a communication device, a processor (for example,digital signal processor (DSP), an image signal processor (ISR), amicrocontroller, a field programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), a central processing unit (CPA), agraphics processing unit (GPU), a visual processing unit (VPU), and soon), possibly with embedded memory, a single core processor, a multicore processor, a core within a processor, any other electroniccomputing device, or any combination of the above.

The operations in accordance with the teachings herein may be performedby a computer specially constructed or programmed to perform thedescribed functions.

As used herein, the phrase “for example,” “such as”, “for instance” andvariants thereof describe non-limiting embodiments of the presentlydisclosed subject matter. Reference in the specification to “one case”,“some cases”, “other cases” or variants thereof means that a particularfeature, structure or characteristic described in connection with theembodiment(s) may be included in at least one embodiment of thepresently disclosed subject matter. Thus the appearance of the phrase“one case”, “some cases”, “other cases” or variants thereof does notnecessarily refer to the same embodiment(s). As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items.

It is appreciated that certain features of the presently disclosedsubject matter, which are, for clarity, described in the context ofseparate embodiments, may also be provided in combination in a singleembodiment. Conversely, various features of the presently disclosedsubject matter, which are, for brevity, described in the context of asingle embodiment, may also be provided separately or in any suitablesub-combination.

The term “image sensor” is recognized by those skilled in the art andrefers to any device configured to capture images, a sequence of images,videos, and so forth. This includes sensors that convert optical inputinto images, where optical input can be visible light (like in acamera), radio waves, microwaves, terahertz waves, ultraviolet light,infrared light, x-rays, gamma rays, and/or any other light spectrum.This also includes both 2D and 3D sensors. Examples of image sensortechnologies may include: CCD, CMOS, NMOS, and so forth. 3D sensors maybe implemented using different technologies, including: stereo camera,active stereo camera, time of flight camera, structured light camera,radar, range image camera, and so forth.

The term “audio sensor” is recognized by those skilled in the art andrefers to any device configured to capture audio data. This includessensors that convert audio and sounds into digital audio data.

In embodiments of the presently disclosed subject matter, one or morestages illustrated in the figures may be executed in a different orderand/or one or more groups of stages may be executed simultaneously andvice versa. The figures illustrate a general schematic of the systemarchitecture in accordance embodiments of the presently disclosedsubject matter. Each module in the figures can be made up of anycombination of software, hardware and/or firmware that performs thefunctions as defined and explained herein. The modules in the figuresmay be centralized in one location or dispersed over more than onelocation.

It should be noted that some examples of the presently disclosed subjectmatter are not limited in application to the details of construction andthe arrangement of the components set forth in the following descriptionor illustrated in the drawings. The invention can be capable of otherembodiments or of being practiced or carried out in various ways. Also,it is to be understood that the phraseology and terminology employedherein is for the purpose of description and should not be regarded aslimiting.

In this document, an element of a drawing that is not described withinthe scope of the drawing and is labeled with a numeral that has beendescribed in a previous drawing may have the same use and description asin the previous drawings.

The drawings in this document may not be to any scale. Different figuresmay use different scales and different scales can be used even withinthe same drawing, for example different scales for different views ofthe same object or different scales for the two adjacent objects.

FIG. 1A is a block diagram illustrating a possible implementation of acommunicating system. In this example, apparatuses 200 a and 200 b maycommunicate with server 300 a, with server 300 b, with cloud platform400, with each other, and so forth. Possible implementations ofapparatuses 200 a and 200 b may include apparatus 200 as described inFIGS. 2A and 2B. Possible implementations of servers 300 a and 300 b mayinclude server 300 as described in FIG. 3. Some possible implementationsof cloud platform 400 are described in FIGS. 4A, 4B and 5. In thisexample apparatuses 200 a and 200 b may communicate directly with mobilephone 111, tablet 112, and personal computer (PC) 113. Apparatuses 200 aand 200 b may communicate with local router 120 directly, and/or throughat least one of mobile phone 111, tablet 112, and personal computer (PC)113. In this example, local router 120 may be connected with acommunication network 130. Examples of communication network 130 mayinclude the Internet, phone networks, cellular networks, satellitecommunication networks, private communication networks, virtual privatenetworks (VPN), and so forth. Apparatuses 200 a and 200 b may connect tocommunication network 130 through local router 120 and/or directly.Apparatuses 200 a and 200 b may communicate with other devices, such asservers 300 a, server 300 b, cloud platform 400, remote storage 140 andnetwork attached storage (NAS) 150, through communication network 130and/or directly.

FIG. 1B is a block diagram illustrating a possible implementation of acommunicating system. In this example, apparatuses 200 a, 200 b and 200c may communicate with cloud platform 400 and/or with each other throughcommunication network 130. Possible implementations of apparatuses 200a, 200 b and 200 c may include apparatus 200 as described in FIGS. 2Aand 2B. Some possible implementations of cloud platform 400 aredescribed in FIGS. 4A, 4B and 5.

FIGS. 1A and 1B illustrate some possible implementations of acommunication system. In some embodiments, other communication systemsthat enable communication between apparatus 200 and server 300 may beused. In some embodiments, other communication systems that enablecommunication between apparatus 200 and cloud platform 400 may be used.In some embodiments, other communication systems that enablecommunication among a plurality of apparatuses 200 may be used.

FIG. 2A is a block diagram illustrating a possible implementation ofapparatus 200. In this example, apparatus 200 may comprise: one or morememory units 210, one or more processing units 220, and one or morecommunication modules 230. In some implementations, apparatus 200 maycomprise additional components, while some components listed above maybe excluded.

FIG. 2B is a block diagram illustrating a possible implementation ofapparatus 200. In this example, apparatus 200 may comprise: one or morememory units 210, one or more processing units 220, one or morecommunication modules 230, one or more power sources 240, one or moreaudio sensors 250, one or more image sensors 260, one or more lightsources 265, one or more motion sensors 270, and one or more positioningsensors 275. In some implementations, apparatus 200 may compriseadditional components, while some components listed above may beexcluded. For example, in some implementations apparatus 200 may alsocomprise at least one of the following: one or more barometers; one ormore pressure sensors; one or more proximity sensors; one or moreelectrical impedance sensors; one or more electrical voltage sensors;one or more electrical current sensors; one or more user input devices;one or more output devices; and so forth. In another example, in someimplementations at least one of the following may be excluded fromapparatus 200: memory units 210, communication modules 230, powersources 240, audio sensors 250, image sensors 260, light sources 265,motion sensors 270, and positioning sensors 275.

In some embodiments, one or more power sources 240 may be configured to:power apparatus 200; power server 300; power cloud platform 400; and/orpower computational node 500. Possible implementation examples of powersources 240 may include: one or more electric batteries; one or morecapacitors; one or more connections to external power sources; one ormore power convertors; any combination of the above; and so forth.

In some embodiments, the one or more processing units 220 may beconfigured to execute software programs. For example, processing units220 may be configured to execute software programs stored on the memoryunits 210. In some cases, the executed software programs may storeinformation in memory units 210. In some cases, the executed softwareprograms may retrieve information from the memory units 210. Possibleimplementation examples of the processing units 220 may include: one ormore single core processors, one or more multicore processors; one ormore controllers; one or more application processors; one or more systemon a chip processors; one or more central processing units; one or moregraphical processing units; one or more neural processing units; anycombination of the above; and so forth.

In some embodiments, the one or more communication modules 230 may beconfigured to receive and transmit information. For example, controlsignals may be transmitted and/or received through communication modules230. In another example, information received though communicationmodules 230 may be stored in memory units 210. In an additional example,information retrieved from memory units 210 may be transmitted usingcommunication modules 230. In another example, input data may betransmitted and/or received using communication modules 230. Examples ofsuch input data may include: input data inputted by a user using userinput devices; information captured using one or more sensors; and soforth. Examples of such sensors may include: audio sensors 250; imagesensors 260; motion sensors 270; positioning sensors 275; chemicalsensors; temperature sensors; barometers; pressure sensors; proximitysensors; electrical impedance sensors; electrical voltage sensors;electrical current sensors; and so forth.

In some embodiments, the one or more audio sensors 250 may be configuredto capture audio by converting sounds to digital information. Someexamples of audio sensors 250 may include: microphones, unidirectionalmicrophones, bidirectional microphones, cardioid microphones,omnidirectional microphones, onboard microphones, wired microphones,wireless microphones, any combination of the above, and so forth. Insome examples, the captured audio may be stored in memory units 210. Insome additional examples, the captured audio may be transmitted usingcommunication modules 230, for example to other computerized devices,such as server 300, cloud platform 400, computational node 500, and soforth. In some examples, processing units 220 may control the aboveprocesses. For example, processing units 220 may control at least oneof: capturing of the audio; storing the captured audio; transmitting ofthe captured audio; and so forth. In some cases, the captured audio maybe processed by processing units 220. For example, the captured audiomay be compressed by processing units 220; possibly followed: by storingthe compressed captured audio in memory units 210; by transmitted thecompressed captured audio using communication modules 230; and so forth.In another example, the captured audio may be processed using speechrecognition algorithms. In another example, the captured audio may beprocessed using speaker recognition algorithms.

In some embodiments, the one or more image sensors 260 may be configuredto capture visual information by converting light to: images; sequenceof images; videos; and so forth. In some examples, the captured visualinformation may be stored in memory units 210. In some additionalexamples, the captured visual information may be transmitted usingcommunication modules 230, for example to other computerized devices,such as server 300, cloud platform 400, computational node 500, and soforth. In some examples, processing units 220 may control the aboveprocesses. For example, processing units 220 may control at least oneof: capturing of the visual information; storing the captured visualinformation; transmitting the captured visual information; and so forth.In some cases, the captured visual information may be processed byprocessing units 220. For example, the captured visual information maybe compressed by processing units 220; possibly followed: by storing thecompressed captured visual information in memory units 210; bytransmitted the compressed captured visual information usingcommunication modules 230; and so forth. In another example, thecaptured visual information may be processed in order to: detectobjects, detect events, detect action, detect face, detect people,recognize person, and so forth.

In some embodiments, the one or more light sources 265 may be configuredto emit light, for example in order to enable better image capturing byimage sensors 260. In some examples, the emission of light may becoordinated with the capturing operation of image sensors 260. In someexamples, the emission of light may be continuous. In some examples, theemission of light may be performed at selected times. The emitted lightmay be visible light, infrared light, x-rays, gamma rays, and/or in anyother light spectrum.

In some embodiments, the one or more motion sensors 270 may beconfigured to perform at least one of the following: detect motion ofobjects in the environment of apparatus 200; measure the velocity ofobjects in the environment of apparatus 200; measure the acceleration ofobjects in the environment of apparatus 200; detect motion of apparatus200; measure the velocity of apparatus 200; measure the acceleration ofapparatus 200; and so forth. In some implementations, the one or moremotion sensors 270 may comprise one or more accelerometers configured todetect changes in proper acceleration and/or to measure properacceleration of apparatus 200. In some implementations, the one or moremotion sensors 270 may comprise one or more gyroscopes configured todetect changes in the orientation of apparatus 200 and/or to measureinformation related to the orientation of apparatus 200. In someimplementations, motion sensors 270 may be implemented using imagesensors 260, for example by analyzing images captured by image sensors260 to perform at least one of the following tasks: track objects in theenvironment of apparatus 200; detect moving objects in the environmentof apparatus 200; measure the velocity of objects in the environment ofapparatus 200; measure the acceleration of objects in the environment ofapparatus 200; measure the velocity of apparatus 200, for example bycalculating the egomotion of image sensors 260; measure the accelerationof apparatus 200, for example by calculating the egomotion of imagesensors 260; and so forth. In some implementations, motion sensors 270may be implemented using image sensors 260 and light sources 265, forexample by implementing a LIDAR using image sensors 260 and lightsources 265. In some implementations, motion sensors 270 may beimplemented using one or more RADARs. In some examples, informationcaptured using motion sensors 270: may be stored in memory units 210,may be processed by processing units 220, may be transmitted and/orreceived using communication modules 230, and so forth.

In some embodiments, the one or more positioning sensors 275 may beconfigured to obtain positioning information of apparatus 200, to detectchanges in the position of apparatus 200, and/or to measure the positionof apparatus 200. In some examples, positioning sensors 275 may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. In someexamples, information captured using positioning sensors 275 may bestored in memory units 210, may be processed by processing units 220,may be transmitted and/or received using communication modules 230, andso forth.

In some embodiments, the one or more chemical sensors may be configuredto perform at least one of the following: measure chemical properties inthe environment of apparatus 200; measure changes in the chemicalproperties in the environment of apparatus 200; detect the present ofchemicals in the environment of apparatus 200; measure the concentrationof chemicals in the environment of apparatus 200. Examples of suchchemical properties may include: pH level, toxicity, temperature, and soforth. Examples of such chemicals may include: electrolytes, particularenzymes, particular hormones, particular proteins, smoke, carbondioxide, carbon monoxide, oxygen, ozone, hydrogen, hydrogen sulfide, andso forth. In some examples, information captured using chemical sensorsmay be stored in memory units 210, may be processed by processing units220, may be transmitted and/or received using communication modules 230,and so forth.

In some embodiments, the one or more temperature sensors may beconfigured to detect changes in the temperature of the environment ofapparatus 200 and/or to measure the temperature of the environment ofapparatus 200. In some examples, information captured using temperaturesensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more barometers may be configured todetect changes in the atmospheric pressure in the environment ofapparatus 200 and/or to measure the atmospheric pressure in theenvironment of apparatus 200. In some examples, information capturedusing the barometers may be stored in memory units 210, may be processedby processing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more pressure sensors may be configuredto perform at least one of the following: detect pressure in theenvironment of apparatus 200; measure pressure in the environment ofapparatus 200; detect change in the pressure in the environment ofapparatus 200; measure change in pressure in the environment ofapparatus 200; detect pressure at a specific point and/or region of thesurface area of apparatus 200; measure pressure at a specific pointand/or region of the surface area of apparatus 200; detect change inpressure at a specific point and/or area; measure change in pressure ata specific point and/or region of the surface area of apparatus 200;measure the pressure differences between two specific points and/orregions of the surface area of apparatus 200; measure changes inrelative pressure between two specific points and/or regions of thesurface area of apparatus 200. In some examples, information capturedusing the pressure sensors may be stored in memory units 210, may beprocessed by processing units 220, may be transmitted and/or receivedusing communication modules 230, and so forth.

In some embodiments, the one or more proximity sensors may be configuredto perform at least one of the following: detect contact of a solidobject with the surface of apparatus 200; detect contact of a solidobject with a specific point and/or region of the surface area ofapparatus 200; detect a proximity of apparatus 200 to an object. In someimplementations, proximity sensors may be implemented using imagesensors 260 and light sources 265, for example by emitting light usinglight sources 265, such as ultraviolet light, visible light, infraredlight and/or microwave light, and detecting the light reflected fromnearby objects using image sensors 260 to detect the present of nearbyobjects. In some examples, information captured using the proximitysensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more electrical impedance sensors may beconfigured to perform at least one of the following: detect change overtime in the connectivity and/or permittivity between two electrodes;measure changes over time in the connectivity and/or permittivitybetween two electrodes; capture Electrical Impedance Tomography (EIT)images. In some examples, information captured using the electricalimpedance sensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more electrical voltage sensors may beconfigured to perform at least one of the following: detect and/ormeasure voltage between two electrodes; detect and/or measure changesover time in the voltage between two electrodes. In some examples,information captured using the electrical voltage sensors may be storedin memory units 210, may be processed by processing units 220, may betransmitted and/or received using communication modules 230, and soforth.

In some embodiments, the one or more electrical current sensors may beconfigured to perform at least one of the following: detect and/ormeasure electrical current flowing between two electrodes; detect and/ormeasure changes over time in the electrical current flowing between twoelectrodes. In some examples, information captured using the electricalcurrent sensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more user input devices may beconfigured to allow one or more users to input information. In someexamples, user input devices may comprise at least one of the following:a keyboard, a mouse, a touch pad, a touch screen, a joystick, amicrophone, an image sensor, and so forth. In some examples, the userinput may be in the form of at least one of: text, sounds, speech, handgestures, body gestures, tactile information, and so forth. In someexamples, the user input may be stored in memory units 210, may beprocessed by processing units 220, may be transmitted and/or receivedusing communication modules 230, and so forth.

In some embodiments, the one or more user output devices may beconfigured to provide output information to one or more users. In someexamples, such output information may comprise of at least one of:notifications, feedbacks, reports, and so forth. In some examples, useroutput devices may comprise at least one of: one or more audio outputdevices; one or more textual output devices; one or more visual outputdevices; one or more tactile output devices; and so forth. In someexamples, the one or more audio output devices may be configured tooutput audio to a user, for example through: a headset, a set ofspeakers, and so forth. In some examples, the one or more visual outputdevices may be configured to output visual information to a user, forexample through: a display screen, an augmented reality display system,a printer, a LED indicator, and so forth. In some examples, the one ormore tactile output devices may be configured to output tactilefeedbacks to a user, for example through vibrations, through motions, byapplying forces, and so forth. In some examples, the output may beprovided: in real time, offline, automatically, upon request, and soforth. In some examples, the output information may be read from memoryunits 210, may be provided by a software executed by processing units220, may be transmitted and/or received using communication modules 230,and so forth.

FIG. 3 is a block diagram illustrating a possible implementation ofserver 300. In this example, server 300 may comprise: one or more memoryunits 210, one or more processing units 220, one or more communicationmodules 230, and one or more power sources 240. In some implementations,server 300 may comprise additional components, while some componentslisted above may be excluded. For example, in some implementationsserver 300 may also comprise at least one of the following: one or moreuser input devices; one or more output devices; and so forth. In anotherexample, in some implementations at least one of the following may beexcluded from server 300: memory units 210, communication modules 230,and power sources 240.

FIG. 4A is a block diagram illustrating a possible implementation ofcloud platform 400. In this example, cloud platform 400 may comprisecomputational node 500 a, computational node 500 b, computational node500 c and computational node 500 d. In some examples, a possibleimplementation of computational nodes 500 a, 500 b, 500 c and 500 d maycomprise server 300 as described in FIG. 3. In some examples, a possibleimplementation of computational nodes 500 a, 500 b, 500 c and 500 d maycomprise computational node 500 as described in FIG. 5.

FIG. 4B is a block diagram illustrating a possible implementation ofcloud platform 400. In this example, cloud platform 400 may comprise:one or more computational nodes 500, one or more shared memory modules410, one or more power sources 240, one or more node registrationmodules 420, one or more load balancing modules 430, one or moreinternal communication modules 440, and one or more externalcommunication modules 450. In some implementations, cloud platform 400may comprise additional components, while some components listed abovemay be excluded. For example, in some implementations cloud platform 400may also comprise at least one of the following: one or more user inputdevices; one or more output devices; and so forth. In another example,in some implementations at least one of the following may be excludedfrom cloud platform 400: shared memory modules 410, power sources 240,node registration modules 420, load balancing modules 430, internalcommunication modules 440, and external communication modules 450.

FIG. 5 is a block diagram illustrating a possible implementation ofcomputational node 500. In this example, computational node 500 maycomprise: one or more memory units 210, one or more processing units220, one or more shared memory access modules 510, one or more powersources 240, one or more internal communication modules 440, and one ormore external communication modules 450. In some implementations,computational node 500 may comprise additional components, while somecomponents listed above may be excluded. For example, in someimplementations computational node 500 may also comprise at least one ofthe following: one or more user input devices; one or more outputdevices; and so forth. In another example, in some implementations atleast one of the following may be excluded from computational node 500:memory units 210, shared memory access modules 510, power sources 240,internal communication modules 440, and external communication modules450.

In some embodiments, internal communication modules 440 and externalcommunication modules 450 may be implemented as a combined communicationmodule, such as communication modules 230. In some embodiments, onepossible implementation of cloud platform 400 may comprise server 300.In some embodiments, one possible implementation of computational node500 may comprise server 300. In some embodiments, one possibleimplementation of shared memory access modules 510 may comprise usinginternal communication modules 440 to send information to shared memorymodules 410 and/or receive information from shared memory modules 410.In some embodiments, node registration modules 420 and load balancingmodules 430 may be implemented as a combined module.

In some embodiments, the one or more shared memory modules 410 may beaccessed by more than one computational node.

Therefore, shared memory modules 410 may allow information sharing amongtwo or more computational nodes 500. In some embodiments, the one ormore shared memory access modules 510 may be configured to enable accessof computational nodes 500 and/or the one or more processing units 220of computational nodes 500 to shared memory modules 410. In someexamples, computational nodes 500 and/or the one or more processingunits 220 of computational nodes 500, may access shared memory modules410, for example using shared memory access modules 510, in order toperform at least one of: executing software programs stored on sharedmemory modules 410, store information in shared memory modules 410,retrieve information from the shared memory modules 410.

In some embodiments, the one or more node registration modules 420 maybe configured to track the availability of the computational nodes 500.In some examples, node registration modules 420 may be implemented as: asoftware program, such as a software program executed by one or more ofthe computational nodes 500; a hardware solution; a combined softwareand hardware solution; and so forth. In some implementations, noderegistration modules 420 may communicate with computational nodes 500,for example using internal communication modules 440. In some examples,computational nodes 500 may notify node registration modules 420 oftheir status, for example by sending messages: at computational node 500startup; at computational node 500 shutdowns; at constant intervals; atselected times; in response to queries received from node registrationmodules 420; and so forth. In some examples, node registration modules420 may query about computational nodes 500 status, for example bysending messages: at node registration module 420 startup; at constantintervals; at selected times; and so forth.

In some embodiments, the one or more load balancing modules 430 may beconfigured to divide the work load among computational nodes 500. Insome examples, load balancing modules 430 may be implemented as: asoftware program, such as a software program executed by one or more ofthe computational nodes 500; a hardware solution; a combined softwareand hardware solution; and so forth. In some implementations, loadbalancing modules 430 may interact with node registration modules 420 inorder to obtain information regarding the availability of thecomputational nodes 500. In some implementations, load balancing modules430 may communicate with computational nodes 500, for example usinginternal communication modules 440. In some examples, computationalnodes 500 may notify load balancing modules 430 of their status, forexample by sending messages: at computational node 500 startup; atcomputational node 500 shutdowns; at constant intervals; at selectedtimes; in response to queries received from load balancing modules 430;and so forth. In some examples, load balancing modules 430 may queryabout computational nodes 500 status, for example by sending messages:at load balancing module 430 startup; at constant intervals; at selectedtimes; and so forth.

In some embodiments, the one or more internal communication modules 440may be configured to receive information from one or more components ofcloud platform 400, and/or to transmit information to one or morecomponents of cloud platform 400. For example, control signals and/orsynchronization signals may be sent and/or received through internalcommunication modules 440. In another example, input information forcomputer programs, output information of computer programs, and/orintermediate information of computer programs, may be sent and/orreceived through internal communication modules 440. In another example,information received though internal communication modules 440 may bestored in memory units 210, in shared memory units 410, and so forth. Inan additional example, information retrieved from memory units 210and/or shared memory units 410 may be transmitted using internalcommunication modules 440. In another example, input data may betransmitted and/or received using internal communication modules 440.Examples of such input data may include input data inputted by a userusing user input devices.

In some embodiments, the one or more external communication modules 450may be configured to receive and/or to transmit information. Forexample, control signals may be sent and/or received through externalcommunication modules 450. In another example, information receivedthough external communication modules 450 may be stored in memory units210, in shared memory units 410, and so forth. In an additional example,information retrieved from memory units 210 and/or shared memory units410 may be transmitted using external communication modules 450. Inanother example, input data may be transmitted and/or received usingexternal communication modules 450. Examples of such input data mayinclude: input data inputted by a user using user input devices;information captured from the environment of apparatus 200 using one ormore sensors; and so forth. Examples of such sensors may include: audiosensors 250; image sensors 260; motion sensors 270; positioning sensors275; chemical sensors; temperature sensors; barometers; pressuresensors; proximity sensors; electrical impedance sensors; electricalvoltage sensors; electrical current sensors; and so forth.

FIG. 6 illustrates an exemplary embodiment of memory 600 storing aplurality of modules. In some examples, memory 600 may be separate fromand/or integrated with memory units 210, separate from and/or integratedwith memory units 410, and so forth. In some examples, memory 600 may beincluded in a single device, for example in apparatus 200, in server300, in cloud platform 400, in computational node 500, and so forth. Insome examples, memory 600 may be distributed across several devices.Memory 600 may store more or fewer modules than those shown in FIG. 6.In this example, memory 600 may comprise: one or more datasets 610, oneor more annotations 620, one or more views 630, one or more algorithms640, one or more tasks 650, one or more logs 660, one or more policies670, one or more permissions 680, and an execution manager module 690.Execution manager module 690 may be implemented in software, hardware,firmware, a mix of any of those, or the like. For example, if themodules are implemented in software, they may contain softwareinstructions for execution by at least one processing device, such asprocessing unit 220, by apparatus 200, by server 300, by cloud platform400, by computational node 500, and so forth. In some examples,execution manager module 690 may be configured to perform at least oneof processes 700, 900, 1100, 1300, 1500, 1700, and so forth.

In some embodiments, dataset 610 may comprise data and information. Forexample, dataset 610 may comprise information pertinent to a subject, anissue, a topic, a problem, a task, and so forth. In some embodiments,dataset 610 may comprise one or more tables, such as database tables,spreadsheets, matrixes, and so forth. In some examples, dataset 610 maycomprise one or more n-dimensional tables, such as tensors. In someembodiments, dataset 610 may comprise information about relations amongitems, for example in a form of graphs, hyper-graphs, lists ofconnections, matrices holding similarities, n-dimensional tables holdingsimilarities, matrices holding distances, n-dimensional tables holdingdissimilarities, and so forth. In some embodiments, dataset 610 maycomprise hierarchical information, for example in the form a tree,hierarchical database, and so forth. In some embodiments, dataset 610may comprise textual information, for example in the form of strings ofcharacters, textual documents, documents in a markup language (such asHTML and XML), and so forth. In some embodiments, dataset 610 maycomprise visual information, such as images, videos, graphical content,and so forth. In some embodiments, dataset 610 may comprise audio data,such as sound recordings, audio recordings, synthesized audio, and soforth.

In some embodiments, dataset 610 may comprise sensor readings, such asaudio captured using audio sensors 250, images captured using imagesensors 260, motion information captured using motion sensors 270,positioning information captured using positioning sensors 275,atmospheric pressure information captured using barometers, pressureinformation captured using pressure sensors, proximity informationcaptured using proximity sensors, electrical impedance informationcaptured using electrical impedance sensors, electrical voltageinformation captured using electrical voltage sensors, electricalcurrent information captured using electrical current sensors, userinput obtained using user input devices, and so forth.

In some embodiments, dataset 610 may comprise data and informationarranged in data-points. For example, a data-point may correspond to anindividual, to an object, to a geographical location, to a geographicalregion, to a species, and so forth. For example, dataset 610 maycomprise a table, and each row or slice may represent a data-point. Forexample, dataset 610 may comprise several tables, and each data-pointmay correspond to entries in one or more tables. For example, adata-point may comprise a text document, a portion of a text document, acorpus of text documents, and so forth. For example, a data-point maycomprise an image, a portion of an image, a video clip, a portion of avideo clip, a group of images, a group of video clips, a time spanwithin a video recording, a sound recording, a time span within a soundrecording, and so forth. For example, a data-point may comprise to agroup of sensor readings. In some examples, dataset 610 may furthercomprise information about relations among data-points, for example adata-point may correspond to a node in a graph or in a hypergraph, andan edge or a hyperedge may correspond to a relation among data-pointsand may be labeled with properties of the relation. In some examples,data-points may be arranged in hierarchies, for example a data-point maycorrespond to a node in a tree.

In some embodiments, a dataset 610 may be produced and/or maintain by asingle user, by multiple users collaborating to produce and/or maintaindataset 610, by an automatic process, by multiple automatic processescollaborating to produce and/or maintain dataset 610, by one or moreusers and one or more automatic processes collaborating to produceand/or maintain dataset 610, and so forth. In some examples, a userand/or an automatic process may produce and/or maintain no dataset 610,a single dataset 610, multiple datasets 610, and so forth.

In some embodiments, annotations 620 may comprise information related todatasets 610 and/or to elements within datasets 610. In some examples, asingle annotation 620 may comprise information related to one dataset orto multiple datasets, and a single dataset 610 may have no, a single, ormultiple annotations related to it. For example, dataset 610 may havemultiple annotations 620 that complement each other, multipleannotations 620 that are inconsistent or contradict each other, and soforth.

In some embodiments, annotation 620 may be produced and/or maintain by asingle user, by multiple users collaborating to produce and/or maintainannotation 620, by an automatic process, by multiple automatic processescollaborating to produce and/or maintain annotation 620, by one or moreusers and one or more automatic processes collaborating to produceand/or maintain annotation 620, and so forth. In some examples, a userand/or an automatic process may produce and/or maintain no annotation620, a single annotation 620, multiple annotations 620, and so forth.

In some examples, annotation 620 may comprise auxiliary informationrelated to datasets 610. In some examples, annotation 620 may comprisehistoric information related to dataset 610. Such historic informationmay include information related to the source of the dataset and/or ofparts of the dataset, historic usages of the dataset and/or of parts ofthe dataset, and so forth. In some examples, annotation 620 may compriseinformation about the dataset and/or about items (such as data-points)in the dataset that is not included in the dataset.

In some embodiments, annotation 620 may comprise labels and/or tagscorresponding to data-points of dataset 610. In some examples, a labelmay comprise an assignment of one value from a list of possible valuesto a data-point. In some examples, a tag may comprise an assignment ofany number of values (including zero, one, two, three, etc.) from a listof possible values to a data-point. For example, the list of possiblevalues may contain types (such as mammal, fish, amphibian, reptile andbird), and a label may assign a single type to a data-point (forexample, fish label may indicate that the data-point describes an animalthat is a fish), while a tag may assign multiple types to a data-point(for example, bird and mammal tags may indicate that the data-pointcomprise a picture of two animals, one bird and one mammal). In someexamples, a label may comprise an assignment of a value from a range ofpossible values to a data-point. For example, a label with a value of195.3 may indicate that the data-point describes a subject weighing195.3 pounds. In some examples, a tag may comprise an assignment of anynumber of values (including zero, one, two, three, etc.) from a range ofpossible values to a data-point. For example, tags with values of 74,73.8 and 74.6 may indicate varying results produced by repeatedmeasurements.

In some embodiments, annotation 620 may comprise desired outputcorresponding to data-points of dataset 610. In some examples, thedesired output may include a picture and/or a video clip. For example, adata-point may include a picture and/or a video clip, and the desiredoutput may include the picture and/or video clip after some processing,such as noise removal, super-resolution, and so forth. In some examples,the desired output may include a mapping. For example, a data-point mayinclude a picture and/or a video clip, and the desired output mayinclude a mapping of pixels and/or regions of the picture and/or videoclip to desired segments. In another example, a data-point may includeaudio data, and the desired output may include a mapping of portions ofthe audio data to segments. In some examples, the desired output mayinclude audio data. For example, a data-point may include audio data,and the desired output may include the audio data after some processing,such as noise removal, source separation, and so forth. In someexamples, the desired output may include processed data. For example, adata-point may include data captured using one or more sensors, and thedesired output may include the data after some processing, such as noiseremoval, convolution, down-sampling, interpolation, and so forth. Insome examples, the desired output may include textual information. Forexample, a data-point may include a picture and/or a video clip, and thedesired output may comprise a textual description of the picture and/orvideo clip. In another example, a data-point may include audio data, andthe desired output may comprise a transcription of the audio data. Inyet another example, a data-point may include textual information, andthe desired output may comprise a synopsis of the textual information.

In some examples, annotation 620 may comprise information arranged invectors and/or tables. For example, each entry in the vector and/or rowin a table and/or column in the table may correspond to a data-point ofdataset 610, and the entry may comprise annotation related to thatdata-point. In some examples, annotation 620 may comprise informationarranged in one or more matrixes. For example, each entry in the matrixmay correspond to two data-points of dataset 610 according to the rowand column of the entry, and the entry may comprise information relatedto these data-points. In some examples, annotation 620 may compriseinformation arranged in one or more tensors. For example, each entry inthe tensor may correspond to a number of data-points of dataset 610according to the indices of the entry, and the entry may compriseinformation related to these data-points. In some examples, annotation620 may comprise information arranged in one or more graphs and/or oneor more hypergraphs. For example, each node in the graph may correspondto a data-point of dataset 610, and an edge of the graph and/orhyperedge of the hypergraph may comprise information related to thedata-points connected by the edges and/or hyperedge.

In some embodiments, view 630 may comprise data and information relatedto datasets 610 and/or annotations 620. In some examples, view 630 maycomprise modified versions of one or more datasets of datasets 610and/or modify versions of one or more annotations of annotations 620.Unless otherwise stated, it is appreciated that any operation discussedwith reference to datasets 610 and/or annotations 620, may also beimplemented in a similar manner with respect to views 630.

In some examples, view 630 may comprise a unification of one or moredatasets of datasets 610. For example, view 630 may comprise a mergingrule for merging two or more datasets. In another example, datasets 610may comprise database tables, and view 630 may comprise SQL expressionsfor generating a new table out of the original tables and/or generatedtable. In yet another example, datasets 610 may comprise data-points,and view 630 may comprise a rule for merging data-points, a rule forselecting a subset of the data-points, and so forth.

In some embodiments, view 630 may comprise a unification of one or moreannotations of annotations 620. For example, view 630 may comprise amerging rule for merging two or more annotations. In another example,annotations 620 may comprise database tables containing annotationinformation, and view 630 may comprise SQL expressions for generating anew annotation table out of the original tables and/or generated table.In yet another example, annotations 620 may comprise informationcorresponding to data-points, and view 630 may comprise a rule formerging the information corresponding to a data-point to obtain newannotation information. Such rule may prioritize information from oneannotation source over others, may include a decision mechanism toproduce new annotation and/or select an annotation out of the originalannotations, and so forth. In another example, annotation 620 maycomprise information corresponding to data-points, and view 630 maycomprise a rule for selecting information corresponding to a subset ofthe data-points.

In some embodiments, view 630 may comprise a selection of one or moredatasets of datasets 610 and one or more annotations of annotations 620.In some examples, view 630 may comprise a selection of one or moredatasets 610 and of a unification of one or more annotations 620, asdescribed above. In some examples, view 630 may comprise a selection ofa unification of one or more datasets of datasets 610 (as describedabove) and of one or more annotations of annotations 620. In someexamples, view 630 may comprise a selection of a unification of one ormore datasets of datasets 610 and of a unification of one or moreannotations of annotations 620. In some examples, view 630 may comprisea selection of one or more other views of views 630.

In some embodiments, algorithms 640 may comprise algorithms forprocessing information, such as the information contained in datasets610 and/or annotations 620 and/or views 630 and/or tasks 650 and/or logs660 and/or policies 670 and/or permissions 680. In some cases,algorithms 640 may further comprise parameters and/or hyper-parametersof the algorithms. For example, algorithms 640 may comprise a pluralityof versions of the same core algorithm with different sets of parametersand/or hyper-parameters.

In some embodiments, algorithm 640 may comprise one or more decisionrules. For example, a decision rule may compare a computed value to athreshold, and in some cases the threshold may be set based on aparameter and/or a hyper-parameter. In some embodiments, algorithm 640may be preprogrammed manually. For example, a manually preprogrammedalgorithm may implement a heuristic algorithm that has zero or moreparameters and/or hyper-parameters. In some embodiments, algorithm 640may comprise a machine learning algorithm configured to train ontraining examples, such as training examples included in datasets 610and/or views 630, to estimate labels and/or tags and/or desired results,such as labels and/or tags and/or desired results included inannotations 620 and/or views 630. For example, algorithm 640 maycomprise a kernel based algorithm, such as support vector machine and/orkernel principal component analysis, and the selection of a kernel maybe according to a hyper-parameter. For example, algorithm 640 maycomprise an artificial neural network, and the structure and/or othercharacteristics of the artificial neural network may be selectedaccording to hyper-parameters. For example, algorithm 640 may comprise aclustering and/or a segmentation algorithm, and the number of desiredclusters and/or segments may be selected according to a hyper-parameter.For example, algorithm 640 may comprise a factorization algorithm, andthe number of desired factors may be determined according to ahyper-parameter. For example, a stopping condition of algorithm 640 maybe based on hyper-parameters. In some examples, algorithm 640 maycomprise a regression algorithm, a classification algorithm, aclustering algorithm, a segmentation algorithm, a factorizationalgorithm, a source separation algorithm, a dimensional reductionalgorithm, a feature selection algorithm, and so forth. In someexamples, algorithms 640 may comprise an algorithm that combines two ormore other algorithms into a single algorithm.

In some embodiments, algorithms 640 may comprise algorithms forprocessing information and data from an external source. In someexamples, the external data source may include a sensor (such as audiosensor, image sensor, motion sensor, positioning sensor, etc.), a user,an external device, an automatic process, external data repository, andso forth. Some examples of external data repositories may include apublic database, a blockchain, a web crawler, and so forth.

In some embodiments, algorithms 640 may process information and datafrom an external source to update datasets 610 and/or annotations 620and/or views 630 and/or algorithms 640 and/or tasks 650 and/or logs 660and/or policies 670 and/or permissions 680. In some examples, based onthe data from the external source, the algorithm may add information toone or more datasets, remove information from one or more datasets,modify information of one or more datasets, and so forth. For example,the algorithm may add new data-points containing the data from theexternal source to a dataset, update existing data-points according tothe data from the external source, remove data-points that were madeobsolete according to data from the external source, and so forth. Insome examples, based on the data from the external source, the algorithmmay add information to one or more annotations, remove information fromone or more annotations, modify information of one or more annotations,and so forth. For example, the algorithm may add new labels and/or tagsand/or desired results containing data from the external source to anannotation, update existing labels and/or tags and/or desired resultsaccording to the data from the external source, remove labels and/ortags and/or desired results that were made obsolete according to datafrom the external source, and so forth. In some examples, algorithms 640may process information and data from an external source to update oneor more data-points of datasets 610 and one or more corresponding labelsand/or tags and/or desired results of annotations 620. In some examples,based on the data from the external source, the algorithm may create anew view, delete a view, modify a view, and so forth. In some examples,the algorithm may create a new view to match an observed phenomenonand/or environment. In some examples, the algorithm may maintain viewsof items relevant to an environment, device, user, problem, and soforth. Some examples of such items may include objects, people, faces,gestures, voices, sounds, and so forth. For example, observing anenvironment with some types of objects may cause the algorithm to createa view containing examples of the observed types, which may later beused to train an object detector. Furthermore, detecting a change insaid environment, such as an introduction of a new type of objects tothe environment or removal of a certain type of objects from theenvironment, may cause the algorithm to modify the view to match thechange. Furthermore, when a situation change in a way that makes theenvironment expendable, the algorithm may delete the corresponding view.For example, observing a conversation of two speakers may cause thealgorithm to create a view containing voice samples of the two speakers.Furthermore, detecting a change in the conversation, for example anintroduction of a new speaker to the conversation, may cause thealgorithm to add voice samples of the new speaker to the view.Furthermore, end of the conversation may cause the algorithm to deletethe corresponding view. In another example, when the data from theexternal source describes a household containing a child and a dog, thealgorithm may create a view containing image examples of children and ofdogs by merging datasets and/or annotations of image examples ofchildren with datasets and/or annotations of image examples of dogs.When the data from the external source indicates an addition of a cat tothe household, image examples of cats may be added to the view bymerging additional datasets and/or annotations of image examples of catsinto the view. Furthermore, when the data from the external sourceindicates that the dog left the household, information related to theimage examples of dogs may be removed from the view.

In some embodiments, algorithms 640 may process information and datafrom datasets 610 and/or annotations 620 and/or views 630 and/oralgorithms 640 and/or tasks 650 and/or logs 660 and/or policies 670and/or permissions 680 to update datasets 610 and/or annotations 620and/or views 630 and/or algorithms 640 and/or tasks 650 and/or logs 660and/or policies 670 and/or permissions 680. For example, based on datafrom some source datasets of datasets 610, the algorithm may update thesource datasets, update other datasets, update annotations associatedwith the source datasets, update other annotations, update views basedon the source datasets, update other views, and so forth. Similarly,based on data from some source annotations of annotations 620, thealgorithm may update the source annotations, update other annotations,update datasets associated with the source annotations, update otherdatasets, update views based on the source annotations, update otherviews, and so forth. Furthermore, based on data from some source viewsof views 630, the algorithm may update the source views, update otherviews, update datasets and/or annotations that the source views arebased on, update other datasets and/or annotations, and so forth.

In some examples, updating datasets 610, for example by an algorithmprocessing data from datasets 610 and/or annotations 620 and/or views630 as described above, may comprise adding new datasets to datasets 610(for example where a new dataset may comprise data-points containingresults of calculations based on the processed information, data-pointsselected from other datasets based on the results of the calculations,etc.), removing datasets from datasets 610 (for example removingdatasets that were made obsolete according to a calculation based on theprocessed information), modifying some of the datasets of datasets 610,and so forth. In some examples, modifying a dataset, for example by analgorithm processing data from datasets 610 and/or annotations 620and/or views 630 as described above, may comprise adding new data-pointsto the dataset (for example where a new data-point contains a result ofa calculation based on the processed information), modifying data-pointsof the dataset (for example changing a value of a data-point accordingto a calculation based on the processed information), removingdata-points from the dataset (for example removing data-points that weremade obsolete according to a calculation based on the processedinformation), and so forth.

In some examples, updating an annotation, for example by an algorithmprocessing data from datasets 610 and/or annotations 620 and/or views630 as described above, may comprise adding new annotations toannotations 620 (for example where a new annotation may comprise tagsand/or labels and/or desired results containing results of calculationsbased on the processed information, tags and/or labels and/or desiredresults selected from other annotations based on the results of thecalculations, etc.), removing annotations from annotations 620 (forexample removing annotations that were made obsolete according to acalculation based on the processed information), modifying some of theannotations of annotations 620, and so forth. In some examples,modifying an annotation may comprise adding tags and/or labels and/ordesired results to the annotation (for example where the value of thelabel and/or tag and/or desired result is according to a result of acalculation based on the processed information), removing tags and/orlabels and/or desired results from the annotation (for example removinglabels and/or tags and/or desired results that were made obsoleteaccording to the calculation's result), modifying tags and/or labelsand/or desired results in the annotation (for example according to thecalculation's result), and so forth. For example, the algorithm may addand/or modify and/or remove labels and/or tags and/or desired results ofthe annotation that corresponds to data-points of datasets 610 based ona result of a calculation that is based on values of said data-points.In another example, the algorithm may produce a new annotation based onseveral other source annotations that deal with the same dataset, forexample using a voting mechanism. In some examples, the algorithm mayupdate the produced annotation in response to a change in the sourceannotations, may delete the produced annotation in response a deletionof one of the source annotations, and so forth.

In some examples, updating a view, for example by an algorithmprocessing data from datasets 610 and/or annotations 620 and/or views630 as described above, may comprise adding new views to views 630,removing views from views 630, modifying some of the views of views 630,and so forth. For example, observing a dataset and/or an annotation withsome distribution of elements may cause the algorithm to create a viewcontaining a sample of the elements with a different distribution.Furthermore, detecting a change of the dataset and/or annotation thatcaused a change of said distribution may lead the algorithm to modifythe view to match the change. Furthermore, when changes to the datasetand/or the annotation cause the said distribution to reach a certaindesired property, the algorithm may remove the view.

In some embodiments, algorithms 640 may process information and datafrom datasets 610 and/or one or more annotations 620 and/or one or moreviews 630 and/or algorithms 640 and/or tasks 650 and/or logs 660 and/orpolicies 670 and/or permissions 680 to update one or more algorithms640. In some examples, based on the processed information, the algorithmmay create a new algorithm, delete an algorithm, modify an algorithm,and so forth. For example, observing a dataset and/or an annotationand/or a view with some distribution of elements may cause the algorithmto create a new algorithm with a set of hyper-parameters matching thedistribution. Furthermore, detecting a change to the dataset and/orannotation and/or view that caused a change in said distribution maycause the algorithm to modify the set of hyper-parameters to match thechanged distribution. Furthermore, when changes to the dataset and/orthe annotation and/or the view cause the said distribution to reach acertain desired property, the algorithm may delete the createdalgorithm.

In some embodiments, tasks 650 may comprise one or more tasks waitingfor execution. In some examples, a task may comprise a selection of oneor more datasets of datasets 610, of one or more annotations ofannotations 620, of one or more views of views 630, of one or morealgorithms of algorithms 640, and so forth. For example, a task maycomprise a selection of a dataset and an algorithm, and the execution ofthe task may comprise applying the selected algorithm to the data of theselected dataset. In another example, a task may comprise a selection ofa dataset, an annotation and an algorithm, and the execution of the taskmay comprise applying the selected algorithm to the data of the selecteddataset and the selected annotation. In yet another example, a task maycomprise a selection of an annotation and an algorithm, and theexecution of the task may comprise applying the selected algorithm toinformation included in the selected annotation. In another example, atask may comprise a selection of a view and an algorithm, and executionof the task may comprise applying the selected algorithm to the selectedview.

In some embodiments, tasks 650 may comprise scheduling information. Forexample, the scheduling information may specify priorities assigned tothe tasks, in the form of a priority values assigned to the tasks (suchas high, medium, low, numerical priority values, etc.), in the form ofan ordering of the tasks according to ascending or descending priority,and so forth. In some examples, the scheduling information may specifypreferred execution times, constraints on execution times, preferredexecution frequencies, constraints on execution frequencies, and soforth.

In some embodiments, tasks 650 may comprise execution requirements. Insome examples, the execution requirements may comprise details aboutpreferred execution conditions and/or settings of the tasks. Forexample, execution requirements may comprise a specification of hardwarerequirements (such as memory size, processing power, etc.) of a task. Inanother example, execution requirements may comprise an indication ofspecific devices and/or specific device types required to execute thetask (such as apparatus 200, server 300, cloud platform 400,computational node 500, and so forth). In yet another example, executionrequirements may comprise a specification of software requirements of atask (such as operation system, software libraries, software modules,and so forth).

In some embodiments, tasks 650 may comprise tasks for manual execution.In some examples, tasks for manual execution may include indications ofspecific persons required to execute the task, of types of personsrequired to execute the task, of skills required to execute the task,and so forth.

In some embodiments, algorithms 640 may process information and datafrom an external source to update one or more tasks of tasks 650. Insome examples, based on the data from the external source, the algorithmmay create a new task, delete a task, modify a task, and so forth. Forexample, in response to an observed change in an environment, thealgorithm may create a new task dealing with datasets and/or annotationsand/or views related to the environment and/or the observed change, mayupdate a task related to the environment and/or the observed change, maydelete a task that became obsolete by the observed change, and so forth.

In some embodiments, algorithms 640 may process information and datafrom datasets 610 and/or annotations 620 and/or views 630 and/oralgorithms 640 and/or tasks 650 and/or logs 660 and/or policies 670and/or permissions 680 to update tasks 650. In some examples, based onthe data from datasets 610 and/or annotations 620 and/or views 630and/or algorithms 640 and/or tasks 650 and/or logs 660 and/or policies670 and/or permissions 680, the algorithm may create a new task, deletea task, modify a task, and so forth. For example, observing an update toa dataset and/or to an annotation and/or to a view and/or to analgorithm may cause the algorithm to create a new task dealing with theupdated items, may cause the algorithm to modify a task dealing with theupdated items, may cause the algorithm to remove a task that was madeobsolete by the update, and so forth.

In some embodiments, logs 660 may comprise historic information relatedto datasets 610, annotations 620, views 630, algorithms 640, tasks 650,logs 660, policies 670, permissions 680, and so forth. In some examples,a log may comprise historic information about creations and/or deletionsand/or modifications of datasets 610, elements of a dataset, annotations620, elements of an annotation, views 630, elements of a view,algorithms 640, core algorithms of an algorithm, parameters and/orhyper-parameters of an algorithm, tasks 650, details of a task, logs660, a log, entries of a log, policies 670, a policy, an element of apolicy, permissions 680, a permission record, and so forth.

In some examples, a log may comprise historic information about pastexecution details of algorithms 640 and/or tasks 650. For example, suchexecution details may include the execution time, the executionduration, the executing device, the executing person, the actualresource requirement, software versions of software used in theexecution, and so forth. For example, such execution details maycomprise errors and/or results of the execution and/or statistics basedon the results of the execution. Furthermore, such execution details maycomprise versions of datasets 610, annotations 620, views 630,algorithms 640, tasks 650, logs 660, policies 670, permissions 680,etc., used in the execution.

In some embodiments, policies 670 may comprise a specification of whatactions need to be taken, for example periodically and/or in response toa change. In some examples, execution manager module 690 may actaccording to policies 670. In some examples, based on policies 670,execution manager module 690 may create and/or delete and/or modifydatasets 610, a dataset, elements of a dataset, annotations 620, anannotation, elements of an annotation, views 630, a view, elements of aview, algorithms 640, an algorithm, core algorithms of an algorithm,parameters and/or hyper-parameters of an algorithm, tasks 650, a task,details of a task, logs 660, elements of a log, policies 670, a policy,a detail of a policy, permissions 680, a permission record, and soforth.

In some embodiments, permissions 680 may specify which user and/orautomatic process and/or algorithm (such as an algorithm of algorithms640) and/or task (such as a task of tasks 650) may access and/or createand/or modify and/or delete which item (such as datasets 610, a dataset,elements of a dataset, annotations 620, an annotation, elements of anannotation, views 630, a view, elements of a view, algorithms 640, analgorithm, core algorithms of an algorithm, parameters and/orhyper-parameters of an algorithm, tasks 650, a task, details of a task,logs 660, a log, elements of a log, policies 670, a policy, a detail ofa policy, permissions 680, a permission record, details of a permissionrecord, and so forth).

In some embodiments, datasets 610 and/or annotations 620 and/or views630 and/or algorithms 640 and/or tasks 650 and/or logs 660 and/orpolicies 670 and/or permissions 680 may be created and/or deleted and/ormodified manually and/or automatically.

In some embodiments, a method, such as methods 700, 900, 1100, 1300,1500 and 1700, may comprise of one or more steps. In some examples, amethod, as well as all individual steps therein, may be performed byvarious aspects of apparatus 200, server 300, cloud platform 400,computational node 500, and so forth. For example, the method may beperformed by processing units 220 executing software instructions storedwithin memory units 210 and/or within shared memory modules 410. In someexamples, a method, as well as all individual steps therein, may beperformed by a dedicated hardware. In some examples, computer readablemedium (such as a non-transitory computer readable medium) may storedata and/or computer implementable instructions for carrying out amethod. Some examples of possible execution manners of a method mayinclude continuous execution (for example, returning to the beginning ofthe method once the method normal execution ends), periodicallyexecution, executing the method at selected times, execution upon thedetection of a trigger (some examples of such trigger may include atrigger from a user, a trigger from another method, a trigger from anexternal device, etc.), and so forth.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples, for example in the cases described below. Someexamples of such machine learning algorithms may include classificationalgorithms, data regressions algorithms, image segmentation algorithms,visual detection algorithms (such as object detectors, face detectors,person detectors, motion detectors, edge detectors, etc.), visualrecognition algorithms (such as face recognition, person recognition,object recognition, etc.), speech recognition algorithms, mathematicalembedding algorithms, natural language processing algorithms, supportvector machines, random forests, nearest neighbors algorithms, deeplearning algorithms, artificial neural network algorithms, convolutionalneural network algorithms, recursive neural network algorithms, linearalgorithms, non-linear algorithms, ensemble algorithms, and so forth.For example, a trained machine learning algorithm may comprise aninference model, such as a predictive model, a classification model, aregression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recursive neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters are set manually by a person or automaticallyby an process external to the machine learning algorithm (such as ahyper parameter search algorithm), and the parameters of the machinelearning algorithm are set by the machine learning algorithm accordingto the training examples. In some implementations, the hyper-parametersare set according to the training examples and the validation examples,and the parameters are set according to the training examples and theselected hyper-parameters.

In some embodiments, trained machine learning algorithms (also referredto as trained machine learning models in the present disclosure) may beused to analyze inputs and generates outputs, for example in the casesdescribed below. In some examples, a trained machine learning algorithmmay be used as an inference model that when provided with an inputgenerates an inferred output. For example, a trained machine learningalgorithm may include a classification algorithm, the input may includea sample, and the inferred output may include a classification of thesample (such as an inferred label, an inferred tag, and so forth). Inanother example, a trained machine learning algorithm may include aregression model, the input may include a sample, and the inferredoutput may include an inferred value for the sample. In yet anotherexample, a trained machine learning algorithm may include a clusteringmodel, the input may include a sample, and the inferred output mayinclude an assignment of the sample to at least one cluster. In anadditional example, a trained machine learning algorithm may include aclassification algorithm, the input may include an image, and theinferred output may include a classification of an item depicted in theimage. In yet another example, a trained machine learning algorithm mayinclude a regression model, the input may include an image, and theinferred output may include an inferred value for an item depicted inthe image (such as an estimated property of the item, such as size,volume, age of a person depicted in the image, cost of a productdepicted in the image, and so forth). In an additional example, atrained machine learning algorithm may include an image segmentationmodel, the input may include an image, and the inferred output mayinclude a segmentation of the image. In yet another example, a trainedmachine learning algorithm may include an object detector, the input mayinclude an image, and the inferred output may include one or moredetected objects in the image and/or one or more locations of objectswithin the image. In some examples, the trained machine learningalgorithm may include one or more formulas and/or one or more functionsand/or one or more rules and/or one or more procedures, the input may beused as input to the formulas and/or functions and/or rules and/orprocedures, and the inferred output may be based on the outputs of theformulas and/or functions and/or rules and/or procedures (for example,selecting one of the outputs of the formulas and/or functions and/orrules and/or procedures, using a statistical measure of the outputs ofthe formulas and/or functions and/or rules and/or procedures, and soforth).

In some embodiments, artificial neural networks may be configured toanalyze inputs and generate corresponding outputs. Some examples of suchartificial neural networks may comprise shallow artificial neuralnetworks, deep artificial neural networks, feedback artificial neuralnetworks, feed forward artificial neural networks, autoencoderartificial neural networks, probabilistic artificial neural networks,time delay artificial neural networks, convolutional artificial neuralnetworks, recurrent artificial neural networks, long short term memoryartificial neural networks, and so forth. In some examples, anartificial neural network may be configured manually. For example, astructure of the artificial neural network may be selected manually, atype of an artificial neuron of the artificial neural network may beselected manually, a parameter of the artificial neural network (such asa parameter of an artificial neuron of the artificial neural network)may be selected manually, and so forth. In some examples, an artificialneural network may be configured using a machine learning algorithm. Forexample, a user may select hyper-parameters for the an artificial neuralnetwork and/or the machine learning algorithm, and the machine learningalgorithm may use the hyper-parameters and training examples todetermine the parameters of the artificial neural network, for exampleusing back propagation, using gradient descent, using stochasticgradient descent, using mini-batch gradient descent, and so forth. Insome examples, an artificial neural network may be created from two ormore other artificial neural networks by combining the two or more otherartificial neural networks into a single artificial neural network, forexample as described below in relation to FIGS. 10C, 10D, 10E, 10F and10G.

FIG. 7 illustrates an example of a method 700 for presenting inferencemodels based on interrelationships among inference models. In thisexample, method 700 may comprise: obtaining information related toinference models (Step 710); obtaining interrelationships information(Step 720); using the interrelationships information to determineinformation related to a first inference model (Step 730); and using thedetermined information related to the first inference model to presentthe first inference model (Step 740). In some implementations, method700 may comprise one or more additional steps, while some of the stepslisted above may be modified or excluded. For example, in some casesStep 710 and/or Step 740 may be excluded from method 700. In someimplementations, one or more steps illustrated in FIG. 7 may be executedin a different order and/or one or more groups of steps may be executedsimultaneously and/or a plurality of steps may be combined into singlestep and/or a single step may be broken down to a plurality of steps.

In some embodiments, in Step 710 information related to a group of atleast three inference models may be obtained. For example, theinformation related to the group of at least three inference models maybe read from memory (for example from memory unit 210, shared memorymodule 410, etc.), received through a communication network (such ascommunication network 130) using a communication device (such ascommunication module 230), received from an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.), and so forth. In some examples,information related to a group of at least three inference modelsobtained by Step 710 may include and/or be based on parameters and/orhyper-parameters of the inference models. In some examples, in Step 710the information related to the group of at least three inference modelsmay be generated. For example, inference models may be used to generateoutputs corresponding to one or more selected inputs, and the generatedoutputs may be analyzed to determine the information related to thegroup of at least three inference models. Some examples of such analysisof the generated outputs may include statistical analysis of theoutputs, comparison of the generated outputs with desired outputs, andso forth. In another example, information related to the inference modelmay be analyzed to determine the information related to the group of atleast three inference models. For example, each inference model mayinclude an artificial neural network, and properties of the structure ofthe artificial neural network (such as number of layers, number ofartificial neurons in selected layers, type of artificial neurons,activation functions, etc.) may be used to determine the informationrelated to the group of at least three inference models. In someexamples, in Step 710 the information related to the group of at leastthree inference models may be obtained from a single source or frommultiple sources. For example, information related to one or moreinference models of the group of at least three inference models may bereceived from a first external device, and information related to one ormore other inference models of the group of at least three inferencemodels may be received from a second external device. In anotherexample, information related to one or more inference models of thegroup of at least three inference models may be received from anexternal device, and information related to one or more other inferencemodels of the group of at least three inference models may be read frommemory. In yet another example, information related to one or moreinference models of the group of at least three inference models may bereceived from an external device and/or read from memory, andinformation related to one or more other inference models of the groupof at least three inference models may be generated as described above.

In some embodiments, in Step 720 a plurality of interrelationshiprecords may be obtained. For example, each interrelationship record maycorrespond to one subgroup of the group of at least three inferencemodels of Step 710, and in some examples each such subgroup may compriseat least two inference models. Some examples of such subgroups mayinclude subgroups of exactly two inference models, of exactly threeinference models, of at least two inference models, of at least threeinference models, of at least six inference models, and so forth. Forexample, one or more interrelationship records may be read from memory(for example from memory unit 210, shared memory module 410, etc.),received through a communication network (such as communication network130) using a communication device (such as communication module 230),received from an external device (such as mobile phone 111, tablet 112,PC 113, remote storage 140, NAS 150, server 300, cloud platform 400,etc.), generated by analyzing the corresponding subgroup of inferencemodels, and so forth. In some examples, one or more interrelationshiprecords may be generated. For example, a machine learning model may betrained using training examples to generate interrelationship recordsfrom two or more inference models, and the trained machine learningmodel may be used to analyze a subgroup of the group of at least threeinference models of Step 710 and generate a correspondinginterrelationship record. Such training example may include a pluralityof inference models together with a desired interrelationship record forthe plurality of inference models. In another example, an artificialneural network may be configured to generate interrelationship recordsfrom two or more inference models, and the artificial neural network maybe used to analyze a subgroup of the group of at least three inferencemodels of Step 710 and generate a corresponding interrelationshiprecord. In yet another example, Step 720 may obtain an interrelationshiprecord corresponding to a subgroup of the group of at least threeinference models of Step 710 comprising a measure of similarity (or ameasure of dissimilarity) among the inference models of the subgroup.For example, the measure of similarity (or the measure of dissimilarity)may comprise a measure of similarity (or a measure of dissimilarity)among properties of the inference models of the subgroup. In anadditional example, Step 720 may obtain an interrelationship recordcorresponding to a subgroup of the group of at least three inferencemodels of Step 710 comprising a statistical measure (such as mean,median, mode, standard deviation, variance, function of an histogram,and so forth) of a property of the inference models. Some examples ofsuch properties are described throughout this disclosure.

In some examples, in Step 720 structures associated with a plurality ofinference models may be compared, for example using an inexact graphmatching algorithm, to determine a matching score related to theplurality of inference models. For example, in Step 720 a structureassociated with the first inference model and a structure associatedwith a second inference model may be compared, for example using aninexact graph matching algorithm, to determine a matching score relatedto the first inference model and the second inference model. Further, insome examples, the determined matching score may be used to determinethe information related to the first inference model, for example byStep 730 as described below. For example, each inference model maycomprise an artificial neural network, and the network graphs of theartificial neural networks of a plurality of inference models may becompared by Step 720 using an inexact graph matching algorithm todetermine a matching score related to the plurality of inference models.In another example, each inference model may comprise an hierarchicalmodel (such as hierarchical clustering model, hierarchicalclassification model, etc.), and the tree graphs of the hierarchicalmodels of a plurality of inference models may be compared by Step 720using an inexact tree matching algorithm to determine a matching scorerelated to the plurality of inference models.

In some examples, in Step 720 a first subgroup of the group of at leastthree inference models may be analyzed to generate the interrelationshiprecord corresponding to the first subgroup, for example as describedabove. Further, in some examples, the generated interrelationship recordcorresponding to the first subgroup may be used to determine theinformation related to the first inference model, for example by Step730 as described below.

In some embodiments, in Step 730 the plurality of interrelationshiprecords obtained by Step 720 may be used to determine informationrelated to a first inference model of the group of at least threeinference models. For example, a machine learning model may be trainedusing training examples to determine information related to a singleinference model from a plurality of interrelationship records, and thetrained machine learning model may be used to analyze the plurality ofinterrelationship records obtained by Step 720 and determine informationrelated to the first inference model. Such training example may includea plurality of interrelationship records, possibly with informationabout the first inference model, together with desired determinedinformation related to the first inference model. In another example, anartificial neural network may be configured to determine informationrelated to a single inference model from a plurality ofinterrelationship records, and the artificial neural network may be usedto analyze the plurality of interrelationship records obtained by Step720 and determine information related to the first inference model. Insome examples, in Step 730 an interrelationship record corresponding toa subgroup of the group of at least three inference models of Step 710,where the subgroup may comprise exactly two inference models, may beused to determine the information related to the first inference model,for example as described above. In some examples, in Step 730 aninterrelationship record corresponding to a subgroup of the group of atleast three inference models of Step 710, where the subgroup maycomprise at least three inference models (for example, exactly three,exactly four, exactly five, exactly six, more than six, etc.), may beused to determine the information related to the first inference model,for example as described above. In some examples, in Step 730 aninterrelationship record corresponding to a subgroup of the group of atleast three inference models of Step 710 may comprise a measure ofsimilarity (or a measure of dissimilarity) as described above, and themeasure of similarity (or the measure of dissimilarity) may be used todetermine the information related to the first inference model. In someexamples, in Step 730 an interrelationship record corresponding to asubgroup of the group of at least three inference models of Step 710 maycomprise a statistical measure of a property of the inference models asdescribed above, and the statistical measure may be used to determinethe information related to the first inference model, for example asdescribed above.

In some examples, in Step 730 interrelationship information based on oneor more commonalities among at least two inference models may be used todetermine the information related to the first inference model.

For example, the at least two inference models may comprise identicalparameters and/or hyper-parameters, the values of the parameters and/orhyper-parameters may be compared to find commonalities among the atleast two inference models in the form of parameters and/orhyper-parameters with identical or substantially identical values. Inanother example, each of the at least two inference models may comprisestructure, and the structures may be compared to find commonalitiesamong the structures. For example, each of the at least two inferencemodels may comprise an artificial neural network, and the network graphsof the artificial neural networks of the at least two inference modelsmay be compared using an exact graph matching algorithm to findcommonalities among the graphs. In another example, each of the at leasttwo inference models may comprise a hierarchical model (such ashierarchical clustering model, hierarchical classification model, etc.),and the tree graphs of the hierarchical models of the at least twoinference models may be compared using an exact tree matching algorithmto find commonalities among the trees. Further, in some examples, ameasure of similarity among the at least two inference models may becalculated based on the found commonalities among the at least twoinference models. For example, number and/or amount and/or size of thecommonalities may be used (for example, after normalization, after acomparison with number and/or amount and/or size of distinct features ofthe at least two inference models, etc.) to determine a measure ofsimilarity of the at least two inference models. Further, in someexamples, the measure of similarity of the at least two inference modelsmay be used to determine the information related to the first inferencemodel, for example as described below.

In some examples, in Step 730 interrelationship information related to ameasure of similarity among at least two inference models may be used todetermine the information related to the first inference model. Forexample, a machine learning model may be trained using training examplesto determine information related to a single inference model frommeasures of similarities, and the trained machine learning model may beused to analyze at least the measure of similarity among at least twoinference models and determine information related to the firstinference model. Such training example may include one or more measuresof similarities, together with desired determined information related tothe first inference model. In another example, an artificial neuralnetwork may be configured to determine information related to a singleinference model from measures of similarities, and the artificial neuralnetwork may be used to analyze at least the measure of similarity amongat least two inference models and determine information related to thefirst inference model.

In some examples, in Step 730 interrelationship information related to ameasure based on an interrelationship between a second inference modeland a third inference model may be used to determine the informationrelated to the first inference model (for example as described above),wherein both the second inference model and the third inference modelmay differ from the first inference model.

In some examples, in Step 730 interrelationship information based on anumber of layers associated with the first inference model and a numberof layers associated with a second inference model may be used todetermine the information related to the first inference model. Forexample, a similarity measure between the first inference model and thesecond inference model may be calculated based on the number of layersassociated with the first inference model and the number of layersassociated with the second inference model, and the information relatedto the first inference model may be determined using the calculatedsimilarity measure as described above. In another example, a statisticalmeasure (such as mean, median, mode, standard deviation, variance,function of an histogram, and so forth) of at least the number of layersassociated with the first inference model and the number of layersassociated with the second inference model may be calculated, and theinformation related to the first inference model may be determined usingthe calculated statistical measure as described above.

In some examples, in Step 730 interrelationship information based on atype of at least one layer of the first inference model and a type of atleast one layer of a second inference model may be used to determine theinformation related to the first inference model. For example, asimilarity measure between the first inference model and the secondinference model may be determined based on the type of at least onelayer of the first inference model and the type of at least one layer ofthe second inference model (for example using the two types to select anentry in a matrix that holds the determined similarity measure), and theinformation related to the first inference model may be determined usingthe determined similarity measure as described above.

In some examples, in Step 730 interrelationship information based on anumber of artificial neurons associated with the first inference modeland a number of artificial neurons associated with a second inferencemodel may be used to determine the information related to the firstinference model. For example, a similarity measure between the firstinference model and the second inference model may be calculated basedon the number of artificial neurons associated with the first inferencemodel and the number of artificial neurons associated with the secondinference model, and the information related to the first inferencemodel may be determined using the calculated similarity measure asdescribed above. In another example, a statistical measure (such asmean, median, mode, standard deviation, variance, function of anhistogram, and so forth) of at least the number of artificial neuronsassociated with the first inference model and the number of artificialneurons associated with the second inference model may be calculated,and the information related to the first inference model may bedetermined using the calculated statistical measure as described above.

In some examples, in Step 730 interrelationship information based on atype of at least one activation function of the first inference modeland a type of at least one activation function of a second inferencemodel may be used to determine the information related to the firstinference model. For example, a similarity measure between the firstinference model and the second inference model may be determined basedon the type of at least one activation function of the first inferencemodel and the type of at least one activation function of the secondinference model (for example using the two types to select an entry in amatrix that holds the determined similarity measure), and theinformation related to the first inference model may be determined usingthe determined similarity measure as described above.

In some examples, in Step 730 interrelationship information based on ameasure of non-linearity associated with the first inference model and ameasure of non-linearity associated with a second inference model may beused to determine the information related to the first inference model.For example, an inference model may use a polynomial kernel function,and the measure of non-linearity may be the degree of the polynomialkernel function. In another example, an inference model may use afunction, a nonlinearity measure of the function may be sampled atselected points, for example using |tf(x)+(1−t)f(y)−f(tx+(1−t)y)| forsome select x, y and t triples, and a function of the samplednonlinearity measure (such as mean, median, max, min, etc.) may be usedas the measure of non-linearity. In one example, a similarity measurebetween the first inference model and the second inference model may becalculated based on the measure of non-linearity associated with thefirst inference model and the measure of non-linearity associated withthe second inference model, and the information related to the firstinference model may be determined using the calculated similaritymeasure as described above. In another example, a statistical measure(such as mean, median, mode, standard deviation, variance, function ofan histogram, and so forth) of at least the measure of non-linearityassociated with the first inference model and the measure ofnon-linearity associated with the second inference model may becalculated, and the information related to the first inference model maybe determined using the calculated statistical measure as describedabove.

In some examples, in Step 730 interrelationship information based on aplurality of results produced using the first inference model and aplurality of results produced using a second inference model may be usedto determine the information related to the first inference model. Forexample, the two pluralities of results may be compared to calculated asimilarity measure between the first inference model and the secondinference model, and the information related to the first inferencemodel may be determined using the calculated similarity measure asdescribed above. In another example, each plurality of results may becompared with desired results to obtain values related to the resultsassociated with the corresponding inference model (such as numbersand/or ratios of false positives and/or false negatives and/or truepositives and/or true negatives, accuracy, precision, F score, confusionmatrix, other statistics about classification errors, sum of absoluteerrors, sum of square errors, other statistics about regression errors,etc.), a statistical measure (such as mean, median, mode, standarddeviation, variance, function of an histogram, and so forth) of thevalues related to the results associated with the first inference modeland the values related to the results associated with the secondinference model may be calculated, and the information related to thefirst inference model may be determined using the calculated statisticalmeasure as described above.

In some examples, in Step 730 interrelationship information based on aplurality of confidence levels associated with results produced usingthe first inference model and a plurality of confidence levelsassociated with results produced using a second inference model may beused to determine the information related to the first inference model.For example, the two pluralities of confidence levels may be compared tocalculated a similarity measure between the first inference model andthe second inference model, and the information related to the firstinference model may be determined using the calculated similaritymeasure as described above.

In some examples, in Step 730 the plurality of interrelationship recordsobtained by Step 720 may be used to determine an embedding of inferencemodels (such as the inference models of Step 710) in a mathematicalspace (for example in a one dimensional space, in a two dimensionalspace, in a three dimensional space, in a space with a dimension higherthan three, and so forth). For example, an embedding of at least twoinference models may be determined, of at least three inference modelsmay be determined, of at least five inference models may be determined,of at least ten inference models may be determined, and so forth. Insome examples, a mathematical vector corresponding to an inference modelmay be obtained, and the mathematical vectors corresponding to theinference models of Step 710 may be used to determine the embedding inthe mathematical space of the inference models of Step 710, for exampleusing principal component analysis (PCA), using Fisher's (lineardiscriminant analysis) LDA, using minor component analysis (MCA), usingt-distributed stochastic neighbor embedding (t-SNE), and so forth. Forexample, an inference model may comprise a matrix, and the mathematicalvector corresponding to the inference model may be generated by amathematical vectorization of the matrix. In another example, aninference model may comprise a number of parameters, and the values ofthe parameters may be used to construct the mathematical vectorcorresponding to the inference model. Further, in some examples, thedetermined embedding of the inference models (in the two dimensionalspace, in the three dimensional space, and so forth) may be visuallypresented to a user. For example, determined embedding of the inferencemodels may be presented to the user using a display screen, an augmentedreality display system, a printer, and so forth. In another example, thedetermined embedding of the inference models may be presented to theuser using output 800 described below.

In some examples, in Step 730 the plurality of interrelationship recordsobtained by Step 720 may be used to determine a hierarchical graph ofinference models (such as the inference models of Step 710). Forexample, a hierarchical graph of at least two inference models may bedetermined, of at least three inference models may be determined, of atleast five inference models may be determined, at least ten inferencemodels may be determined, and so forth. For example, theinterrelationship records may indicate that a particular inference modelis comprise of a composition of one or more other inference models (forexample as described below in relation to FIGS. 10C, 10D, 10E, 10F and10G), and as a result the determined hierarchical graph may have theparticular inference model as a parent to the one or more otherinference models. In another example, a hierarchical clusteringalgorithm may be used to analyze the inference models and construct thehierarchical graph. Further, in some examples, the hierarchical graph ofthe at least three inference models may be visually presented to a user.For example, the hierarchical graph of the at least three inferencemodels may be presented to the user using a display screen, an augmentedreality display system, a printer, and so forth.

In some examples, the information related to the first inference modeldetermined by Step 730 may include textual information describing thefirst inference model. For example, the first inference model may beembedded in a mathematical space as described above, and the textualinformation may describe the location of the first inference model inthe mathematical space, the neighboring inference model to the firstinference model in the mathematical space, and so forth. In anotherexample, the first inference model may be included in an hierarchicalgraph of inference models as described above, and the textualinformation may describe the location of the first inference model inthe hierarchical graph, the parent and/or siblings of the firstinference model in the hierarchical graph, and so forth. In yet anotherexample, similar inference models to the first inference model may beidentified by Step 730 based on the plurality of interrelationshiprecords obtained by Step 720, and the textual information may describethe identified similar inference models.

In some embodiments, in Step 740 the information determined by Step 730may be used to present the first inference model to a user. For example,the information determined by Step 730 may be presented to the userusing a display screen, an augmented reality display system, a printer,and so forth. In another example, the information determined by Step 730may be presented to the user using output 800 described below. In yetanother example, the information determined by Step 730 may comprisetextual information as described above, and the textual information maybe provided to the user (for example, visually, audibly, and so forth).In an additional example, the information determined by Step 730 maycomprise an embedding of the inference models in an embedded space, andthe embedded space may be presented to the user. In yet another example,the information determined by Step 730 may comprise a hierarchical graphof inference models, and the hierarchical graph of inference models maybe presented to the user.

FIG. 8 is a schematic illustration of an example output 800 consistentwith an embodiment of the present disclosure. In this example, output800 may include output region 802 and output region 804. In thisexample, output region 802 may include a visual representation of anembedding of six inference models 812, 814, 816, 818, 822 and 824 in atwo dimensional space, and output region 804 may include textualinformation related to inference model 816. In some examples, a user mayprovide an indication of one or more of the inference models representedin output region 802, and textual details for the indicated inferencemodels may be presented. In some examples, textual details of aninference model, such as the textual details presented in 804 forinference model 816, may include indications of other inference models,and in some implementations the user may select to view more details ofone or more of the other inference models, for example by pressing on anindication of the one or more of the other inference models in outputregion 804. In this example, the textual details presented in outputregion 804 for inference model 816 may include a name of the model(“Model ID #1143”), a textual description of the model (“Cat vs. Dogclassifier”), details about the creation of the model (“Trained usingdatasets #492, #8132”), measures of the quality of the model (“F score0.92”), list of similar models together with indications of a degree ofsimilarity (“68% similarity to Model #2475”, “54% similarity to Model#8453”, “19% similarity to Model #958”), suggestions for improvement ofthe model (“Consider combining with Models #979 and #1943 to improveclassification of Pomeranian dogs”), and suggestions for usages of themodel (“Consider combining with Model #93 to identify dog breed”). Inother examples, other details about the model may be presented, such asdetails based on information determined using method 700 as describedabove.

FIG. 9 illustrates an example of a method 900 for personalized selectionof inference models based on usage of other inference models. In thisexample, method 900 may comprise: receiving indications of usage ofinference models by personalized devices (Step 910); selectingadditional inference models based the inference models used byparticular personalized devices (Step 920); and providing informationrelated to the selected additional inference models to the correspondingpersonalized devices (Step 930). In some implementations, method 900 maycomprise one or more additional steps, while some of the steps listedabove may be modified or excluded. For example, in some cases Step 910and/or Step 930 may be excluded from method 900. In someimplementations, one or more steps illustrated in FIG. 9 may be executedin a different order and/or one or more groups of steps may be executedsimultaneously and/or a plurality of steps may be combined into singlestep and/or a single step may be broken down to a plurality of steps.

In some embodiments, in Step 910 indications of usage of inferencemodels by personalized devices may be received. For example, anindication of a usage of a first inference model by a first personalizeddevice may be received, and an indication of a usage of a secondinference model by a second personalized device may be also received.Some examples of such personalized devices may include mobile phone 111,tablet 112, PC 113, virtual machine operating using server 300 and/orcloud platform 400, and so forth. For example, the indications of usageof inference models may be read from memory (for example from memoryunit 210, shared memory module 410, etc.), read from a blockchain,received through a communication network (such as communication network130) using a communication device (such as communication module 230),received from an external device (such as mobile phone 111, tablet 112,PC 113, remote storage 140, NAS 150, server 300, cloud platform 400,etc.), and so forth. In another example, the indications of usage ofinference models may be generated, for example by a process and/or adevice using the inference models. In yet another example, theindications of usage of inference models may be extracted from a log(such as logs 660).

In some embodiments, in Step 920 additional inference models may beselected for particular personalized devices based on the inferencemodels used by the particular personalized devices. For example, a firstadditional inference model may be selected based on the first inferencemodel used by the first personalized device, and a second additionalinference model may be selected based on the second inference model usedby the second personalized device, wherein the second additionalinference model may differ from the first additional inference model.For example, a table (such as a database table, a data-structure inmemory, etc.) may link the inference models being used to selectedadditional inference models. In another example, a machine learningalgorithm may be trained using training examples to select additionalinference models based on inference models being used, and the trainedmachine learning algorithm may be used to select additional inferencemodels based on the inference models being used by a particularpersonalized device. Some examples of such training examples may includepairs of lists, where each pair may comprise one list of the inferencemodels being used and one list of the desired selection of additionalinference models in that training example. In yet another example, anartificial neural network may be configured to select additionalinference models based on inference models being used, and theartificial neural network may be used to select additional inferencemodels based on the inference models being used by a particularpersonalized device.

In some examples, the selection of the additional inference model byStep 920 may be based on a compatibility of an output of the inferencemodel used by a particular personalized device and an input of theadditional inference model. In some examples, a particular personalizeddevice may be configured to use the output of the inference modelcurrently used by the particular personalized device as input to theadditional inference model. For example, the selection of the firstadditional inference model by Step 920 may be based on a compatibilityof an output of the first inference model used by the first personalizeddevice and an input of the first additional inference model, and theselection of the second additional inference model by Step 920 may bebased on a compatibility of an output of the second inference model usedby the second personalized device and an input of the second additionalinference model. In some examples, the output of the inference model maybe compatible with an input of another inference model when the type ofthe output and the type of the input are identical (for example, bothintegers, both numerical, both strings, both images, etc.), whenproperties of the output matches properties of the input (for example,when the range of the possible outputs is included in the range of thevalid inputs, when the size in bits of the two are identical, when thecomponents of the output image are identical to the components of theinput image, when the size of an output image is a valid size for inputimage, etc.), any combination of the above, and so forth. For example, adata-structure and/or a database storing information about the typesand/or properties of the inputs and/or outputs of inference models maybe accessed to determine whether the output of the inference model usedby a particular personalized device and an input of another inferencemodel are compatible. In another example, such data-structure and/ordatabase may be accessed to obtain a list of alternative inferencemodels with input that is compatible to the output of the inferencemodel used by a particular personalized device, and the additionalinference model for the particular personalized device may be selectedof the obtained list of alternative inference models.

In some examples, the selection of the additional inference model byStep 920 may be based on a compatibility of an input of the inferencemodel used by a particular personalized device and an output of theadditional inference model. In some examples, a particular personalizeddevice may be configured to use the output of the additional inferencemodel as input to the inference model currently used by the particularpersonalized device. For example, the selection of the first additionalinference model by Step 920 may be based on a compatibility of an inputof the first inference model used by the first personalized device andan output of the first additional inference model, and the selection ofthe second additional inference model by Step 920 may be based on acompatibility of an input of the second inference model used by thesecond personalized device and an output of the second additionalinference model. For example, a data-structure and/or a database storinginformation about the types and/or properties of the inputs and/oroutputs of inference models may be accessed to determine whether theinput of the inference model used by a particular personalized deviceand an output of another inference model are compatible. In anotherexample, such data-structure and/or database may be accessed to obtain alist of alternative inference models with output that is compatible tothe input of the inference model used by a particular personalizeddevice, and the additional inference model for the particularpersonalized device may be selected of the obtained list of alternativeinference models.

In some examples, the selection of the additional inference model byStep 920 may be based on a compatibility of an input of the inferencemodel used by a particular personalized device and an input of theadditional inference model. In some examples, a particular personalizeddevice may be configured to replace the inference model currently usedby the particular personalized device with the additional inferencemodel, may be configured to start handling some of the inputs previouslyhandled by the currently used inference model with the additionalinference model (for example as described below), to use the currentlyused inference model and the additional inference model in an ensemblemodel, and so forth. For example, the selection of the first additionalinference model by Step 920 may be based on a compatibility of an inputof the first inference model used by the first personalized device andan input of the first additional inference model, and the selection ofthe second additional inference model by Step 920 may be based on acompatibility of an input of the second inference model used by thesecond personalized device and an input of the second additionalinference model. For example, a data-structure and/or a database storinginformation about the types and/or properties of the inputs of inferencemodels may be accessed to determine whether the input of the inferencemodel used by a particular personalized device and an input of anotherinference model are compatible. In another example, such data-structureand/or database may be accessed to obtain a list of alternativeinference models with input that is compatible to the input of theinference model used by a particular personalized device, and theadditional inference model for the particular personalized device may beselected of the obtained list of alternative inference models.

In some examples, the selection of the additional inference model byStep 920 may be based on a compatibility of an output of the inferencemodel used by a particular personalized device and an output of theadditional inference model. In some examples, a particular personalizeddevice may be configured to replace the inference model currently usedby the particular personalized device with the additional inferencemodel, may be configured to start performing some of the functionspreviously performed by the currently used inference model with theadditional inference model (for example as described below), to use thecurrently used inference model and the additional inference model in anensemble model, and so forth. For example, the selection of the firstadditional inference model by Step 920 may be based on a compatibilityof an output of the first inference model used by the first personalizeddevice and an output of the first additional inference model, and theselection of the second additional inference model by Step 920 may bebased on a compatibility of an output of the second inference model usedby the second personalized device and an output of the second additionalinference model. For example, a data-structure and/or a database storinginformation about the types and/or properties of the outputs ofinference models may be accessed to determine whether the output of theinference model used by a particular personalized device and an outputof another inference model are compatible. In another example, suchdata-structure and/or database may be accessed to obtain a list ofalternative inference models with output that is compatible to theoutput of the inference model used by a particular personalized device,and the additional inference model for the particular personalizeddevice may be selected of the obtained list of alternative inferencemodels.

In some examples, the indication of the usage of the inference models bythe personalized devices received by Step 910 may comprise informationrelated to outputs produced by the inference models when used by thepersonalized devices. For example, the indication of the usage of thefirst inference model by the first personalized device received by Step910 may comprise information related to outputs produced by the firstinference model when used by the first personalized device, and theindication of the usage of the second inference model by the secondpersonalized device received by Step 910 may comprise informationrelated to outputs produced by the second inference model when used bythe second personalized device. Further, in some examples, the selectionof the additional inference models by Step 920 may be based on theinformation related to the outputs produced by the inference models whenused by the personalized devices. For example, the selection of thefirst additional inference model by Step 920 may be based on theinformation related to the outputs produced by the first inference modelwhen used by the first personalized device, and the selection of thesecond additional inference model by Step 920 may be based on theinformation related to the outputs produced by the second inferencemodel when used by the second personalized device. For example, theinformation related to outputs produced by an inference model maycomprise the actual outputs, sample of the outputs, statisticalinformation about the outputs (such as histogram of outputs, mean valueand/or median value and/or mode value of the outputs, variance of theoutputs, standard deviation of the outputs, etc.), information about acomparison of the outputs with desired outputs (such as numbers and/orratios of false positives and/or false negatives and/or true positivesand/or true negatives, accuracy, precision, F score, confusion matrix,other statistics about classification errors, sum of absolute errors,sum of square errors, other statistics about regression errors, etc.),and so forth. Further, in some examples, in response to a firstinformation related to the outputs produced by an inference model whenused by a personalized device (such as a first set of actual outputs, afirst sample of the outputs, a first statistical information about theoutputs, a first information about a comparison of the outputs withdesired outputs, etc.), a first additional inference model may beselected, and in response to a second information related to the outputsproduced by an inference model when used by a personalized device (suchas a second set of actual outputs, a second sample of the outputs, asecond statistical information about the outputs, a second informationabout a comparison of the outputs with desired outputs, etc.), a secondadditional inference model may be selected. For example, a machinelearning algorithm may be trained using training examples to selectadditional inference models based on information related to outputs, andthe trained machine learning algorithm may be used by Step 920 to selectthe additional inference models according to the information related tothe outputs produced by the inference models when used by thepersonalized devices. In another example, an artificial neural networkmay be configured to select additional inference models based oninformation related to outputs, and the artificial neural network may beused by Step 920 to select the additional inference models according tothe information related to the outputs produced by the inference modelswhen used by the personalized devices.

In some examples, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may compriseinformation related to classification errors of the first inferencemodel when used by the first personalized device, the indication of theusage of the second inference model by the second personalized devicereceived by Step 910 may comprise information related to classificationerrors of the second inference model when used by the secondpersonalized device, the selection of the first additional inferencemodel by Step 920 may be based on the information related to theclassification errors of the first inference model when used by thefirst personalized device, and the selection of the second additionalinference model by Step 920 may be based on the information related tothe classification errors of the second inference model when used by thesecond personalized device, for example as described above.

In some examples, the indication of the usage of the inference models bythe personalized devices received by Step 910 may comprise informationrelated to confidence levels associated with outputs produced by theinference models when used by the personalized devices. For example, theindication of the usage of the first inference model by the firstpersonalized device received by Step 910 may comprise informationrelated to confidence levels associated with outputs produced by thefirst inference model when used by the first personalized device, theindication of the usage of the second inference model by the secondpersonalized device received by Step 910 may comprise informationrelated to confidence levels associated with outputs produced by thesecond inference model when used by the second personalized device, andso forth. Further, in some examples, the selection of the additionalinference models by Step 920 may be based on the information related tothe confidence levels associated with the outputs produced by theinference models when used by the personalized devices. For example, theselection of the first additional inference model by Step 920 may bebased on the information related to the confidence levels associatedwith the outputs produced by the first inference model when used by thefirst personalized device, and the selection of the second additionalinference model by Step 920 may be based on the information related tothe confidence levels associated with the outputs produced by the secondinference model when used by the second personalized device. Forexample, the information related to confidence levels associated withthe outputs produced by an inference model may comprise the actualconfidence levels, sample of the confidence levels, statisticalinformation about the confidence levels (such as histogram of confidencelevels, mean value and/or median value and/or mode value of theconfidence levels, variance of the confidence levels, standard deviationof the confidence levels, etc.), and so forth. Further, in someexamples, in response to a first information related to the confidencelevels associated with the outputs produced by an inference model whenused by a personalized device (such as a first set of actual confidencelevels, a first sample of the confidence levels, a first statisticalinformation about the confidence levels, etc.), a first additionalinference model may be selected, and in response to a second informationrelated to the confidence levels associated with the outputs produced byan inference model when used by a personalized device (such as a secondset of actual confidence levels, a second sample of the confidencelevels, a second statistical information about the confidence levels,etc.), a second additional inference model may be selected. For example,a machine learning algorithm may be trained using training examples toselect additional inference models based on information related toconfidence levels, and the trained machine learning algorithm may beused by Step 920 to select the additional inference models according tothe information related to the confidence levels associated with theoutputs produced by the inference models when used by the personalizeddevices. In another example, an artificial neural network may beconfigured to select additional inference models based on informationrelated to confidence levels, and the artificial neural network may beused by Step 920 to select the additional inference models according tothe information related to the confidence levels associated with theoutputs produced by the inference models when used by the personalizeddevices.

In some examples, the indication of the usage of the inference models bythe personalized devices received by Step 910 may comprise informationrelated to usage time of the inference models by the personalizeddevices. For example, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may compriseinformation related to usage time of the first inference model by thefirst personalized device, the indication of the usage of the secondinference model by the second personalized device received by Step 910may comprise information related to usage time of the second inferencemodel by the second personalized device, and so forth. For example, theinformation related to usage time of the inference models by thepersonalized devices may comprise at least one of overall usage time,number of usages, particular time of usage, statistical informationabout the particular times of usage, and so forth. Further, in someexamples, the selection of the additional inference models by Step 920may be based on the information related to the usage time of theinference models by the personalized devices. For example, the selectionof the first additional inference model by Step 920 may be based on theinformation related to the usage time of the first inference model bythe first personalized device, and the selection of the secondadditional inference model by Step 920 may be based on the informationrelated to the usage time of the second inference model by the secondpersonalized device, and so forth. For example, in response to a firstinformation related to the usage time of an inference model by apersonalized device (such as a first overall usage time, a first numberof usages, first particular times of usage, first statisticalinformation about the particular times of usage, etc.), a firstadditional inference model may be selected, and in response to a secondinformation related to the usage time of an inference model by apersonalized device (such as a second overall usage time, a secondnumber of usages, second particular time of usage, second statisticalinformation about the particular times of usage, etc.), a secondadditional inference model may be selected. For example, a machinelearning algorithm may be trained using training examples to selectadditional inference models based on information related to usage time,and the trained machine learning algorithm may be used by Step 920 toselect the additional inference models according to the informationrelated to usage time of the inference models by the personalizeddevices. In another example, an artificial neural network may beconfigured to select additional inference models based on informationrelated to usage time, and the artificial neural network may be used byStep 920 to select the additional inference models according to theinformation related to usage time of the inference models by thepersonalized devices.

In some examples, the indication of the usage of the inference models bythe personalized devices received by Step 910 may comprise informationrelated to locations of usage of the inference models by thepersonalized devices. For example, the indication of the usage of thefirst inference model by the first personalized device received by Step910 may comprise information related to locations of usage of the firstinference model by the first personalized device, the indication of theusage of the second inference model by the second personalized devicereceived by Step 910 may comprise information related to location ofusage of the second inference model by the second personalized device.For example, the information related to location of usage of theinference models by the personalized devices may comprise at least oneof particular locations of usage, statistical information about theparticular locations of usage, and so forth. Further, in some examples,the selection of the additional inference models by Step 920 may bebased on the information related to the location of usage of theinference models by the personalized devices. For example, the selectionof the first additional inference model by Step 920 may be based on theinformation related to the location of usage of the first inferencemodel by the first personalized device, and the selection of the secondadditional inference model by Step 920 may be based on the informationrelated to the location of usage of the second inference model by thesecond personalized device. For example, in response to firstinformation related to the location of usage of an inference model by apersonalized device, a first additional inference model may be selected,and in response to second information related to the location of usageof an inference model by a personalized device, a second additionalinference model may be selected. For example, a machine learningalgorithm may be trained using training examples to select additionalinference models based on information related to locations, and thetrained machine learning algorithm may be used by Step 920 to select theadditional inference models according to the information related tolocation of usage of the inference models by the personalized devices.In another example, an artificial neural network may be configured toselect additional inference models based on information related tolocations, and the artificial neural network may be used by Step 920 toselect the additional inference models according to the informationrelated to location of usage of the inference models by the personalizeddevices.

In some examples, the indication of the usage of the inference models bythe personalized devices received by Step 910 may comprise informationrelated to inputs provided to the inference models by the personalizeddevices. For example, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may compriseinformation related to inputs provided to the first inference model bythe first personalized device, the indication of the usage of the secondinference model by the second personalized device received by Step 910may comprise information related to inputs provided to the secondinference model by the second personalized device. For example, theinformation related to inputs produced by an inference model maycomprise the actual inputs, sample of the inputs, statisticalinformation about the inputs (such as histogram of inputs, mean valueand/or median value and/or mode value of the inputs, variance of theinputs, standard deviation of the inputs, etc.), and so forth. Further,in some examples, the selection of the additional inference models byStep 920 may be based on the information related to the inputs providedto the inference models by the personalized devices. For example, theselection of the first additional inference model by Step 920 may bebased on the information related to the inputs provided to the firstinference model by the first personalized device, and the selection ofthe second additional inference model by Step 920 may be based on theinformation related to the inputs provided to the second inference modelby the second personalized device. For example, in response to a firstinformation related to the inputs produced by an inference model whenused by a personalized device (such as a first set of actual inputs, afirst sample of the inputs, a first statistical information about theinputs, etc.), a first additional inference model may be selected, andin response to a second information related to the inputs produced by aninference model when used by a personalized device (such as a second setof actual inputs, a second sample of the inputs, a second statisticalinformation about the inputs, etc.), a second additional inference modelmay be selected. For example, a machine learning algorithm may betrained using training examples to select additional inference modelsbased on information related to inputs, and the trained machine learningalgorithm may be used by Step 920 to select the additional inferencemodels according to the information related to the inputs produced bythe inference models when used by the personalized devices. In anotherexample, an artificial neural network may be configured to selectadditional inference models based on information related to inputs, andthe artificial neural network may be used by Step 920 to select theadditional inference models according to the information related to theinputs produced by the inference models when used by the personalizeddevices.

In some examples, the indication of the usage of the inference models bythe personalized devices received by Step 910 may comprise indicationsof properties of inference models used by the personalized devices. Insome examples, in Step 920 properties of inference models used by thepersonalized devices may be used to select the additional inferencemodels for the personalized devices. For example, properties of thefirst inference model used by the first personalized device may be usedto select the first additional inference model, properties of the secondinference model used by the second personalized device may be used toselect the second additional inference model, and so forth. Someexamples of such properties may include type of input, type of output,complexity, memory requirements, runtime requirements, informationrelated to outputs produced by the inference models when used by thepersonalized devices (for example as described above), informationrelated to errors of the inference models when used by the personalizeddevices (for example as described above), information related toconfidence levels associated with outputs produced by the inferencemodels when used by the personalized devices (for example as describedabove), information related to confidence levels associated with outputsproduced by the inference models when used by the personalized devices(as described above), and so forth. For example, at least one propertyof the first inference model may be used to access a database ofinference model relations to select the first additional inferencemodel, at least one property of the second inference model may be usedto access a database of inference model relations to select the secondadditional inference model, and so forth. For example, a hash value maybe calculated for the properties using a hash function, and thecalculated hash value may be used to access the database. In anotherexample, a value of a property may be used to access the databasedirectly.

In some examples, the additional inference models selected by Step 920based on the inference models used by the personalized devices may besuggested to users of the personalized devices. For example, the firstadditional inference model selected by Step 920 based on the firstinference model used by the first personalized device may be suggestedto a user of the first personalized device, the second additionalinference model selected by Step 920 based on the second inference modelused by the second personalized device may be suggested to a user of thesecond personalized device, and so forth. In some examples, informationrelated to the selected additional inference model may be provided tothe users of the personalized devices. For example, the informationrelated to the selected additional inference model may be displayed tothe user, may be provided audibly, may be provided textually, may beprovided through the corresponding personalized device, may be providedthrough another device (such as another personalized device associatedwith the user, a public device, etc.), and so forth. For example, theinformation related to the selected additional inference model may betransmitted to an external device (for example through a communicationnetwork, such as communication network 130, using a communicationdevice, such as communication module 230), and in some examples thetransmitted information may be configured to cause the external deviceto provide the information to a user. In another example, theinformation related to the selected additional inference model may bestored in memory, and in some examples the stored information may beread from memory (for example by another process, another device, etc.)and used to provide information related to the selected additionalinference model to a user.

In some examples, the first personalized device may be caused to use acascade of the first inference model and the first additional inferencemodel selected by Step 920 based on the first inference model, forexample as described below in relation to FIG. 10C and FIG. 10D.

In some examples, the first personalized device may be caused to use anaggregate of outputs of the first inference model corresponding to aninput and of outputs of the first additional inference modelcorresponding to the input to determine an overall output correspondingto the input, for example as described below in relation to FIG. 10G andComponent 1008.

In some examples, the first personalized device may be caused to providea first input to the first inference model to determine a first outputand/or to provide a second input to the first additional inference modelselected by Step 920 based on the first inference model to determine asecond output, where the second input differs from the first input.Further, in some examples, the first personalized device may be causedto combine the first output and the second output to determine anoverall output corresponding to a combination of the first input and thesecond input. For example, the first personalized device may be causedto use Component 1006 (for example as illustrated in FIG. 10F) todetermine which input and/or which portion of which input to provide towhich inference model, for example as described below (for example inrelation to FIG. 10E, FIG. 10F and Component 1006).

In some examples, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may comprisean indication that the first personalized device uses the firstinference model to handle at least a first type of input and a secondtype of input. Further, in some examples, for example in response to thereceived indication of usage of the first inference model by the firstpersonalized device, the first personalized device may be caused to usethe first inference model to handle at least the first type of input,and use the first additional inference model selected based on the firstinference model to handle at least the second type of input, for exampleas illustrated in FIG. 10F and as described below in relation to FIG.10F and Component 1006. Some examples of such types of input may includeimages of a particular range of pixel resolution, videos of a particularrange of frame rate, data in a particular format, data from particularsources, and so forth.

In some examples, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may comprisean indication that the first personalized device uses the firstinference model to handle at least inputs received at a first timewindow and inputs received at a second time window. Further, in someexamples, for example in response to the received indication of usage ofthe first inference model by the first personalized device, the firstpersonalized device may be caused to use the first inference model tohandle at least inputs received at the first time window, and use thefirst additional inference model selected based on the first inferencemodel to handle at least inputs received at the second time window, forexample as illustrated in FIG. 10F and as described below in relation toFIG. 10F and Component 1006. Some examples of time windows may includeselected range of time in day (such as morning hours, afternoon hours,evening hours, night hours, between 8 AM to 10 AM, etc.), selected rangeof time in week (such as Monday, Monday between 8 AM and 10 AM,weekdays, weekend, etc.), selected range of time in month (first Mondayof each month, second day of the month, etc.), holidays, and so forth.

In some examples, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may comprisean indication that the first personalized device uses the firstinference model to handle at least inputs associated with a firstgeographical area and inputs associated with a second geographical area.Further, in some examples, for example in response to the receivedindication of usage of the first inference model by the firstpersonalized device, the first personalized device may be caused to usethe first inference model to handle at least inputs associated with thefirst geographical area, and use the first additional inference modelselected based on the first inference model to handle at least inputsassociated with the second geographical area, for example as illustratedin FIG. 10F and as described below in relation to FIG. 10F and Component1006. Some examples of such geographical areas may include home of auser, workplace of a user, indoor, outdoor, particular building,particular part of a building, particular street, particularneighborhood, particular city, and so forth.

In some examples, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may comprisean indication that the first personalized device uses the firstinference model to handle at least inputs associated with a first entityand inputs associated with a second entity. Further, in some examples,for example in response to the received indication of usage of the firstinference model by the first personalized device, the first personalizeddevice may be caused to use the first inference model to handle at leastinputs associated with the first entity, and use the first additionalinference model selected based on the first inference model to handle atleast inputs associated with the second entity, for example asillustrated in FIG. 10F and as described below in relation to FIG. 10Fand Component 1006. Some examples of such entities may include users,family members of a user, friends of a user, coworkers of a user,colleges of a user, robots, legal entities, and so forth.

In some examples, the indication of the usage of the first inferencemodel by the first personalized device received by Step 910 may comprisean indication that the first personalized device uses the firstinference model to handle at least inputs associated with a firstcontextual situation and inputs associated with a second contextualsituation. Further, in some examples, for example in response to thereceived indication of usage of the first inference model by the firstpersonalized device, the first personalized device may be caused to usethe first inference model to handle at least inputs associated with thefirst contextual situation, and use the first additional inference modelselected based on the first inference model to handle at least inputsassociated with the second contextual situation, for example asillustrated in FIG. 10F and as described below in relation to FIG. 10Fand Component 1006. Some examples of such contextual situations mayinclude association with one or more of selected locations, selectedtimes, selected topics, selected persons, selected items, selectedscenes, and so forth.

In some embodiments, in Step 930 information related to the additionalinference models selected by Step 920 may be provided to thecorresponding personalized devices, for example in response to theindications of usage of inference models by personalized devicesreceived by Step 910. For example, in response to the indication ofusage of the first inference model by the first personalized devicereceived by Step 910, information related to the first additionalinference model selected by Step 920 based on the first inference modelmay be provided to the first personalized device, and in response to theindication of usage of the second inference model by the secondpersonalized device received by Step 910, information related to thesecond additional inference model selected by Step 920 based on thesecond inference model may be provided to the second personalizeddevice. In some example, Step 930 and providence of the informationrelated to the additional inference models selected by Step 920 may betriggered by the indications of usage of inference models bypersonalized devices received by Step 910, by the selection of theadditional inference models selected by Step 920, at a selected updatetime, and so forth. In some examples, information related to theadditional inference models selected by Step 920 may be stored in memory(for example in memory unit 210, in shared memory module 410, etc.) forthe corresponding personalized devices to access it, posted on ablockchain for the corresponding personalized devices to access it,transmitted through a communication network (such as communicationnetwork 130) using a communication device (such as communication module230) to the corresponding personalized devices, provided to an externaldevice (such as mobile phone 111, tablet 112, PC 113, remote storage140, NAS 150, server 300, cloud platform 400, etc.) for thecorresponding personalized devices to access it through the externaldevices, and so forth.

FIGS. 10A, 10B, 100, 10D, 10E, 10F and 10G are schematic illustrationsof different configurations for using inference models consistent withembodiments of the present disclosure. In some embodiments, differentdevices may use different configurations of inference models. In someembodiments, a device may use different configurations of inferencemodels at different times, at different locations, in differentcontexts, and so forth.

In some embodiments, inference model 1002 may be used to generate outputbased on provided input, as illustrated in FIG. 10A. In some embodimentsadditional inference models 1004 may be selected based on the usage ofinference model 1002, for example using method 900 as described above.In some embodiments, after a selection of additional inference model1004, inference model 1002 may be replaced by the selected additionalinference model 1004, and the selected additional inference model 1004may be used to generate output based on provided input, as illustratedin FIG. 10B.

In some embodiments, after a selection of additional inference model1004, outputs of inference model 1002 may be provided to the selectedadditional inference model 1004, and the selected additional inferencemodel 1004 may generate output based on the outputs of inference model1002, as illustrated in FIG. 10C. In some examples, inference model 1002and the selected additional inference model 1004 may be used in acascade configuration. For example, before providing the outputs ofinference model 1002 to the selected additional inference model 1004,the outputs of inference model 1002 may be analyzed to determine whetherto provide the outputs of inference model 1002 to the selectedadditional inference model 1004 or not. Further, in response to adetermination not to provide the outputs of inference model 1002 to theselected additional inference model 1004, an output may be generatedbased on that determination and/or the outputs of inference model 1002,and in response to a determination to provide the outputs of inferencemodel 1002 to the selected additional inference model 1004, an outputmay be generated by the selected additional inference model 1004 basedon the outputs of inference model 1002.

In some embodiments, after a selection of additional inference model1004, the selected additional inference model 1004 may be used togenerate outputs based on provided inputs, the outputs of the selectedadditional inference model 1004 may be provided to inference model 1002,and inference model 1002 may generate output based on the outputs of theselected additional inference model 1004, as illustrated in FIG. 10D. Insome examples, inference model 1002 and the selected additionalinference model 1004 may be used in a cascade configuration. Forexample, before providing the outputs of the selected additionalinference model 1004 to inference model 1002, the outputs of theselected additional inference model 1004 may be analyzed to determinewhether to provide the outputs of the selected additional inferencemodel 1004 to inference model 1002 or not. Further, in response to adetermination not to provide the outputs of the selected additionalinference model 1004 to inference model 1002, an output may be generatedbased on that determination and/or the outputs of the selectedadditional inference model 1004, and in response to a determination toprovide the outputs of the selected additional inference model 1004 toinference model 1002, an output may be generated by inference model 1002based on the outputs of the selected additional inference model 1004.

In some embodiments, after a selection of additional inference model1004, common inputs may be provided to both inference model 1002 and theselected additional inference model 1004, inference model 1002 maygenerate a first output (“Output1” in FIG. 10E) based on the providedinput, and the selected additional inference model 1004 may generate asecond output (“Output2” in FIG. 10E) based on the provided input, asillustrated in FIG. 10E. In some embodiments, after a selection ofadditional inference model 1004, a first portion of a common input maybe provided to inference model 1002 and a second portion of the commoninput may be provided to the selected additional inference model 1004,inference model 1002 may generate a first output (“Output1” in FIG. 10E)based on the provided first portion of the common input, and theselected additional inference model 1004 may generate a second output(“Output2” in FIG. 10E) based on the provided second portion of thecommon input, as illustrated in FIG. 10E. For example, the first portionand second portion of the common input may be selected based on thestructure of the common input, based on the content of the common input,based on geographical area associated with the common input, based ontime associated with the common input, based on context associated withthe common input, based on an entity associated with the common input,and so forth. For example, a machine learning algorithm may be trainedusing training examples to select portions of the common input based onthe information listed above, and the trained machine learning algorithmmay be used to select the first portion and second portion of the commoninput based on the information listed above. Some examples of suchtraining examples may include example common inputs and/or exampleinformation related to the example common inputs together with a desiredselection of portions. In another example, an artificial neural networkmay be configured to select portions of the common input based on theinformation listed above, and the artificial neural network may be usedto select the first portion and second portion of the common input basedon the information listed above.

In some embodiments, after a selection of additional inference model1004, Component 1006 may determine (for example as described below)whether to provide the input (or selected portions of the input) toinference model 1002, to inference model 1004, to both inference models1002 and 1004, to none of the two inference models, etc., and in someexamples, the inference models that were provided with the input byComponent 1006 may generate outputs based on the provided input, forexample when an input is provided to inference model 1002 by Component1006, inference model 1002 may generate a first output (“Output1” inFIG. 10F) based on the provided input, and when an input is provided tothe selected additional inference model 1004 by Component 1006, theselected additional inference model 1004 may generate a second output(“Output2” in FIG. 10F) based on the provided input, for example asillustrated in FIG. 10F.

In some embodiments, after a selection of additional inference model1004, common inputs (or portions of the common inputs selected asdescribed above) may be provided to both inference model 1002 and theselected additional inference model 1004, inference model 1002 maygenerate a first output (“Output1” in FIG. 10G) based on the providedinput, the selected additional inference model 1004 may generate asecond output (“Output2” in FIG. 10G) based on the provided input, andthe first output (“Output1”) and the second output (“Output2”) may bemerged into a unified output by Component 1008 (for example as describedbelow), for example as illustrated in FIG. 10G.

In some embodiments, Component 1006 and/or Component 1008 may beperformed by various aspects of apparatus 200, server 300, cloudplatform 400, computational node 500, and so forth. For example,processing units 220 may execute software instructions stored withinmemory units 210 and/or within shared memory modules 410, and thesoftware instructions may be configured to cause processing units 220 toperform the function of Component 1006 and/or Component 1008. In anotherexample, Component 1006 and/or Component 1008 may be implemented by adedicated hardware. In yet another example, computer readable medium(such as a non-transitory computer readable medium) may store dataand/or computer implementable instructions for carrying out thefunctions of Component 1006 and/or Component 1008.

In some embodiments, Component 1006 may determine to which inferencemodels to provide the input (or selected portions of the input), forexample as illustrated in FIG. 10F. In some examples, the determinationto which inference models to provide the input (or selected portions ofthe input) may be based on a type of the input (for example as describedabove), may be based on a time associated with the input (for example asdescribed above), may be based on a geographical area associated withthe input (for example as described above), may be based on a contextualsituation associated with the input (for example as described above),may be based on an entity associated with the input (for example asdescribed above), and so forth. In some examples, Component 1006 mayanalyze the input (or selected parts of the input) to determine to whichinference models to provide the input (or selected portions of theinput), and/or to select which portions of the input to provide to whichinference model. For example, a machine learning algorithm may betrained using training examples to determine which inference modelsshould be provided with which portions of an input based on the contentof the input and/or information associated with the input (such as typeof the input, time associated with the input, geographical areaassociated with the input, contextual situation associated with theinput, entity associated with the input, etc.), and the trained machinelearning algorithm may be used to determine to which portion of theinput to provide to which inference models based on the input and/orinformation associated with the input. For example, such trainingexample may comprise an example of an input and/or of informationassociated with the example input, together with desired selection ofinference models to provide the example input to and/or desiredselection of portions of the example input. In another example, anartificial neural network may be configured to determine which inferencemodels should be provided with which portions of an input based on thecontent of the input and/or information associated with the input (suchas type of the input, time associated with the input, geographical areaassociated with the input, contextual situation associated with theinput, entity associated with the input, etc.), and the artificialneural network may be used to determine to which inference models toprovide which portions of the input based on the input and/orinformation associated with the input.

In some embodiments, Component 1008 may merge a plurality of outputsinto a unified output, for example as illustrated in FIG. 10G. In someexamples, Component 1008 may use a voting mechanism to select one outputof the plurality of outputs. For example, such voting mechanism mayselect the most prevalence output of the plurality of outputs. In someexamples, Component 1008 may calculate a statistical measure of theplurality of outputs, and the unified output may be based on thecalculated statistical measure. Some examples of such statisticalmeasure may include mean, median, mode, variance, standard deviation,and so forth. In some examples, each output of the plurality of outputsmay be associated with a confidence level (for example, with aconfidence level produced by the inference model that generated theoutput, with a confidence level that was calculated for the output,etc.), and Component 1008 may use the confidence levels associated withthe outputs to merge the plurality of outputs into a unified output. Forexample, the output corresponding to the highest confidence level may beselected, a weighted average of the plurality of outputs may becalculated using weights based on the confidence levels, a statisticalmeasure of a subgroup of the plurality of outputs selected based on theconfidence levels may be calculated (for example, the subgroup maycorrespond to outputs associated with the highest confidence levels, tooutputs associated with confidence levels higher than a selectedthreshold, etc.), and so forth.

FIG. 11 illustrates an example of a method 1100 for selective update ofinference models. In this example, method 1100 may comprise: obtaining afirst inference model and a second inference model based on a firstversion of a set of training examples (Step 1110); obtaining updatedversion of the set of training examples (Step 1120); generating updatedversion of the first inference model based on the updated version of theset of training examples (Step 1130); evaluating the updated version ofthe first inference model (Step 1140); and determining whether to updatethe second inference model based on the evaluation result (Step 1150).In some implementations, method 1100 may comprise one or more additionalsteps, while some of the steps listed above may be modified or excluded.For example, in some cases Step 1110 and/or Step 1120 and/or Step 1130may be excluded from method 1100. In some implementations, one or moresteps illustrated in FIG. 11 may be executed in a different order and/orone or more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, in Step 1110 inference models based on a firstversion of a set of training examples may be obtained. For example, afirst inference model and a second inference model may be obtained,where the first inference model and the second inference model may bebased on the first version of the set of training examples. For example,the inference models may be a result of training one or more machinelearning algorithms using the first version of the set of trainingexamples. In another example, the inference models may be selected of aplurality of alternative inference models based on the first version ofthe set of training examples. In some examples, the inference modelsand/or information related to the inference models may be read frommemory (for example from memory unit 210, shared memory module 410,etc.), read from a blockchain, received through a communication network(such as communication network 130) using a communication device (suchas communication module 230), received from an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.), generated (for example bytraining one or more machine learning algorithms using one or morehyper-parameters and/or the first version of the set of trainingexamples), selected (for example as described above), and so forth.

In some embodiments, in Step 1120 an updated version of the set oftraining examples may be obtained. For example, the first version of theset of training examples of Step 1110 may include at least one examplenot included in the updated version of the set of training examplesobtained by Step 1120. In another example, the updated version of theset of training examples obtained by Step 1120 may include at least oneexample not included in the first version of the set of trainingexamples of Step 1110. In yet another example, the updated version ofthe set of training examples obtained by Step 1120 may include at leastone modified version of at least one example included in the firstversion of the set of training examples of Step 1110. In some examples,in Step 1120 update information may be obtained, and in some examplesthe set of training examples of Step 1110 may be updated according tothe obtained update information. For example, the update information maycomprise a delta between the first version of the set of trainingexamples and the updated version of the set of training examples, maycomprise training examples to be added to the first version of the setof training examples, may comprise indication of training examples to beremoved from the first version of the set of training examples, mayinclude indication of training examples to be modified, and so forth. Insome examples, in Step 1120 the updated version of the set of trainingexamples may be read from memory (for example from memory unit 210,shared memory module 410, etc.), read from a blockchain, receivedthrough a communication network (such as communication network 130)using a communication device (such as communication module 230),received from an external device (such as mobile phone 111, tablet 112,PC 113, remote storage 140, NAS 150, server 300, cloud platform 400,etc.), generated (for example as described above), and so forth.

In some embodiments, in Step 1130 an updated version of the firstinference model obtained by Step 1110 may be generated based on theupdated version of the set of training examples obtained by Step 1120.In some examples, a machine learning algorithm may be trained using theupdated version of the set of training examples obtained by Step 1120 togenerate the updated version of the first inference model. For example,the machine learning algorithm used to generate the first inferencemodel (for example as described above) may be trained using the updatedversion of the set of training examples obtained by Step 1120 togenerate the updated version of the first inference model. In anotherexample, an additional inference model may be generated and/or obtainedbased on the updated version of the set of training examples obtained byStep 1120 (for example, in a similar fashion as described above inrelation to Step 1110), and the additional inference model may becombined with the first inference model to generate the updated versionof the first inference model, for example as described above andillustrated in FIG. 10C and/or FIG. 10D and/or FIG. 10E and/or FIG. 10Fand/or FIG. 10G. In some examples, the additional inference model and/orthe updated version of the first inference model may be selected of aplurality of alternative inference models based on the updated versionof the set of training examples obtained by Step 1120. For example, inresponse to a first updated version of the set of training examples, oneinference model may be selected, and in response to a second updatedversion of the set of training examples, another inference model may beselected.

In some examples, the first version of the set of training examples ofStep 1110 may include at least one example not included in the updatedversion of the set of training examples obtained by Step 1120, and theat least one example not included in the updated version of the set oftraining examples may be used by Step 1130 to generate the updatedversion of the first inference model. For example, in response to afirst example included in the first version of the set of trainingexamples and not in the updated version of the set of training examples,Step 1130 may generate a first updated version of the first inferencemodel, and in response to a second example included in the first versionof the set of training examples and not in the updated version of theset of training examples, Step 1130 may generate a second updatedversion of the first inference model different from the first updatedversion of the first inference model.

In some examples, the updated version of the set of training examplesobtained by Step 1120 may include at least one example not included inthe first version of the set of training examples of Step 1110, and theat least one example not included in the first version of the set oftraining examples may be used to generate the updated version of thefirst inference model. For example, in response to a first exampleincluded in the updated version of the set of training examples and notin the first version of the set of training examples, Step 1130 maygenerate a first updated version of the first inference model, and inresponse to a second example included in the updated version of the setof training examples and not in the first version of the set of trainingexamples, Step 1130 may generate a second updated version of the firstinference model different from the first updated version of the firstinference model.

In some embodiments, in Step 1140 the updated version of the firstinference model generated by Step 1130 may be evaluated to obtain anevaluation result. In some examples, parameters of the updated versionof the first inference model generated by Step 1130 (such as parametersdetermined by training a machine learning model using hyper-parametersand/or training examples as described above) may be analyzed to evaluatethe updated version of the first inference model generated by Step 1130.For example, a statistical measure of the parameters may be calculated,and the statistical measure may be used to evaluate the updated versionof the first inference model generated by Step 1130. In another example,a machine learning model may be trained using training examples togenerate evaluations of inference models based on parameters of theinference models, and the trained machine learning model may be used toanalyze the parameters of the updated version of the first inferencemodel to evaluate the updated version of the first inference modelgenerated by Step 1130. Some examples of such training example mayinclude a set of parameters together with a corresponding desiredevaluation. In yet another example, an artificial neural network may beconfigured to generate evaluations of inference models based onparameters of the inference models, and the artificial neural networkmay be used to analyze the parameters of the updated version of thefirst inference model to evaluate the updated version of the firstinference model generated by Step 1130. In some examples, the updatedversion of the first inference model generated by Step 1130 may beevaluated using validation examples and/or test examples to obtain theevaluation result, for example as described above. For example, theupdated version of the first inference model generated by Step 1130 maybe used to estimate outputs for the example inputs of the validationexamples and/or test examples, the estimated outputs may be compared tothe corresponding desired outputs, and the evaluation result may bebased on the comparison of the estimated outputs and the desiredoutputs. For example, the evaluation result may be based on a functionof one or more of the numbers and/or ratios of false positives and/orfalse negatives and/or true positives and/or true negatives, accuracy,precision, F score, confusion matrix, other statistics aboutclassification errors, sum of absolute errors, sum of square errors,other statistics about regression errors, and so forth. In someexamples, the updated version of the first inference model generated byStep 1130 may be evaluated based on entities and/or tools involved inthe generation of the updated version of the first inference modelgenerated by Step 1130. For example, an inference model generated by afirst engineer may be evaluated as better than an inference modelgenerated by a second engineer. In another example, an inference modelgenerated using a first tool may be evaluated as better than aninference model generated using a second tool. Some examples of suchtools may include hardware (such as computing devices, memory devices,etc.), algorithms (such as machine learning algorithms, data preparationalgorithms, etc.), hyper-parameters, and so forth. In some examples, theupdated version of the first inference model generated by Step 1130 maybe evaluated based on usage of the updated version of the firstinference model. For example, the evaluation may be based on amount ofusage, time of usage, location of usage, type of usage, and so forth.For example, an inference model that is used more often may be evaluatedas better than an inference model that is used less often. In anotherexample, an inference model that is used during selected time intervalsmay be evaluated as better than an inference model that is used in othertime intervals. In yet another example, an inference model that is usedat selected locations may be evaluated as better than an inference modelthat is used at other locations. In an additional example, an inferencemodel that is used in particulate fashion may be evaluated as betterthan an inference model that is used in a different fashion. In someexamples, the updated version of the first inference model generated byStep 1130 may be evaluated based on feedbacks from users. For example,information based on outputs of the updated version of the firstinference model generated by Step 1130 may be provided to one or moreusers, the one or more users may provide feedbacks relating to theprovided information, and the updated version of the first inferencemodel generated by Step 1130 may be evaluated based on those feedbacks.

In some embodiments, in Step 1150 the evaluation result obtained by Step1140 may be used to determine whether to update the second inferencemodel obtained by Step 1110. For example, in response to a firstevaluation result (for example, a positive evaluation result, anevaluation result above a selected threshold, an evaluation resultwithin a selected range, etc.), it may be determined to update thesecond inference model obtained by Step 1110, and in response to asecond evaluation result (for example, a negative evaluation result, anevaluation result below a selected threshold, an evaluation resultwithin a selected range, etc.), it may be determined not to update thesecond inference model obtained by Step 1110.

In some embodiments, the update to the second inference model obtainedby Step 1110 may be based on the evaluation result obtained by Step1140. For example, in response to a first evaluation result (forexample, a positive evaluation result, an evaluation result above aselected threshold, an evaluation result within a selected range, etc.),a first update to the second inference model obtained by Step 1110 maybe selected, and in response to a second evaluation result (for example,a negative evaluation result, an evaluation result below a selectedthreshold, an evaluation result within a selected range, etc.), a secondupdate to the second inference model obtained by Step 1110 may beselected.

In some examples, the first inference model obtained by Step 1110 may bea result of training a machine learning algorithm using a first trainingset (for example as described above), the second inference modelobtained by Step 1110 may be a result of training the same or adifferent machine learning algorithm using a second training set (forexample as described above), and both the first training set and thesecond training set may comprise the set of training examples of Step1110. For example, the second training set may differ from the firsttraining set. In another example, the second training set may beidentical to the first training set.

In some examples, the first inference model obtained by Step 1110 may bea result of training a first machine learning algorithm using the set oftraining examples of Step 1110 (for example as described above), thesecond inference model obtained by Step 1110 may be a result of traininga second machine learning algorithm using the set of training examplesof Step 1110 (for example as described above), the updated version ofthe first inference model generated by Step 1130 may be a result oftraining the first machine learning algorithm using the updated versionof the set of training examples obtained by Step 1120 (for example asdescribed above), and updating the second inference model may comprisetraining the second machine learning algorithm using the updated versionof the set of training examples obtained by Step 1120 (for example asdescribed above).

In some examples, the first inference model obtained by Step 1110 may bea result of training a machine learning algorithm using the set oftraining examples of Step 1110 and a first hyper-parameter of themachine learning algorithm (for example as described above), the secondinference model obtained by Step 1110 may be a result of training themachine learning algorithm using the set of training examples of Step1110 and a second hyper-parameter of the machine learning algorithm (forexample as described above), the updated version of the first inferencemodel generated by Step 1130 may be a result of training the machinelearning algorithm using the updated version of the set of trainingexamples obtained by Step 1120 and the first hyper-parameter of themachine learning algorithm (for example as described above), andupdating the second inference model may comprise training the machinelearning algorithm using the updated version of the set of trainingexamples obtained by Step 1120 and the second hyper-parameter of themachine learning algorithm (for example as described above).

In some examples, the first inference model obtained by Step 1110 may beused by a first personalized device, and the second inference modelobtained by Step 1110 may be used by a second personalized device.Further, in some examples, in response to a first evaluation result (forexample, a positive evaluation result, an evaluation result above aselected threshold, an evaluation result within a selected range, etc.),the second personalized device may be caused to use an updated versionof the second inference model (for example as described below), and inresponse to a second evaluation result (for example, a negativeevaluation result, an evaluation result below a selected threshold, anevaluation result within a selected range, etc.), causing the secondpersonalized device to use the updated version of the second inferencemodel may be withheld and/or forwent.

In some examples, the first inference model obtained by Step 1110 may beused by a first personalized device, the second inference model obtainedby Step 1110 may be used by a second personalized device. Further, insome examples, the first personalized device may be caused to use theupdated version of the first inference model generated by Step 1130 (forexample as described below). Further, in some examples, results of theusage of the updated version of the first inference model by the firstpersonalized device may be received (for example from the firstpersonalized device, from a third device, through a communicationnetwork using a communication device, through digital memory, and soforth). Further, in some examples, in Step 1140 the received results ofthe usage of the updated version of the first inference model by thefirst personalized device may be used to evaluate the updated version ofthe first inference model and obtain the evaluation result, for exampleas described above.

In some examples, the first inference model and the second inferencemodel obtained by Step 1110 may be used by a first personalized device.Further, in some examples, a second personalized device may be caused touse the updated version of the first inference model generated by Step1130 and/or an updated version of the second inference model, forexample as described below. In some examples, the updated version of theset of training examples obtained by Step 1120 may include trainingexamples that are compatible with the second personalized device, and asa result the updated version of the first inference model generated byStep 1130 and/or an updated version of the second inference model may becompatible with the second personalized device. For example, examplesthat are compatible with the second personalized device may be selectedby Step 1120 of the first version of the set of training examples ofStep 1110, and the updated version of the set of training examples mayinclude the selected examples. In another example, examples that arecompatible with the second personalized device may be added to the firstversion of the set of training examples of Step 1110 by Step 1120 togenerate the updated version of the set of training examples. In yetanother example, examples that are incompatible with the secondpersonalized device may be removed from the first version of the set oftraining examples of Step 1110 by Step 1120 to generate the updatedversion of the set of training examples. In an additional example, oneor more examples of the first version of the set of training examples ofStep 1110 may be transformed by Step 1120 to be more compatible with thesecond personalized device, and the updated version of the set oftraining examples may include the transformed training examples. In someexamples, the first personalized device may include a first set ofsensors and the second personalized device may include a second set ofsensors, examples in the first version of the set of training examplesof Step 1110 may be compatible with the first set of sensors, andexamples in the updated version of the set of training examples may beselected and/or transformed to be compatible with the second set ofsensors. For example, the first set of sensors may include sensors of afirst type and the second set of sensors may include sensors of a secondtype different from the first type (for example, the first type may bean image sensor and the second type may include an audio sensor, thefirst type may be an 2D image sensor and the second type may be a 3Dimage sensor, and so forth). In another example, the first set ofsensors and the second set of sensors may include sensors of the sametype but with different configurations (for example, different capturingparameters). In yet another example, the first set of sensors and thesecond set of sensors may include sensors of the same type with similarconfigurations that are positioned in different environments and/ordifferent locations within the same environment and/or in differentdirections.

In some embodiments, causing a personalized device (such as the firstpersonalized device mentioned above, the second personalized devicementioned above, etc.) to use an inference model (such as the updatedversion of the first inference model generated by Step 1130, the updatedversion of the second inference model mentioned above, and so forth) maycomprise providing information configured to cause the personalizeddevice to use the inference model. For example, such providedinformation may comprise an indication of the inference model, maycomprise an indication of one or more properties of the inference model,may comprise information related to the inference model, may comprise atleast a part of the inference model, may comprise an indication of atime to use the inference model at, may comprise an indication of alocation to use the inference model at, may comprise an indication of acontext to use the inference model at, and so forth. For example, theprovided information may be stored in memory (for example in memory unit210, in shared memory module 410, etc.) for the personalized device toaccess it, posted on a blockchain for the personalized device to accessit, transmitted through a communication network (such as communicationnetwork 130) using a communication device (such as communication module230) to the personalized device, provided to an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.) for the personalized device toaccess it through the external devices, and so forth.

In some examples, the first inference model obtained by Step 1110 may beused to analyze a plurality of inputs and produce a first plurality ofoutputs corresponding to the inputs. Further, in some examples, theupdated version of the first inference model generated by Step 1130 maybe used to analyze the plurality of inputs and produce a secondplurality of outputs corresponding to the inputs. Further, in someexamples, in Step 1140 the first plurality of outputs produced using thefirst inference model and/or the second plurality of outputs producedusing the updated version of the first inference model may be used toevaluate the updated version of the first inference model and obtain theevaluation result. For example, the first plurality of outputs producedusing the first inference model may be compared with the secondplurality of outputs produced using the updated version of the firstinference model, and the evaluation of the updated version of the firstinference model by Step 1140 may be based on a result of the comparison.In another example, the plurality of inputs may be associated withcorresponding desired outputs, the first plurality of outputs producedusing the first inference model may be compared with the correspondingdesired outputs to obtain an intermediate evaluation result of the firstinference model, the second plurality of outputs produced using theupdated version of the first inference model may be compared with thecorresponding desired outputs to obtain an intermediate evaluationresult of the updated version of the first inference model, and theevaluation result may be based on the comparison of the intermediateevaluation result of the first inference model and the intermediateevaluation result of the updated version of the first inference model.Some examples of such intermediate evaluation results may include afunction of one or more of the numbers and/or ratios of false positivesand/or false negatives and/or true positives and/or true negatives,accuracy, precision, F score, confusion matrix, other statistics aboutclassification errors, sum of absolute errors, sum of square errors,other statistics about regression errors, and so forth. In someexamples, a first evaluation may be assigned to an updated version ofthe first inference model in response to a first comparison result, anda second evaluation may be assigned to the updated version of the firstinference model in response to a second comparison result, where thefirst evaluation may be better than the second evaluation.

In some examples, the updated version of the first inference modelgenerated by Step 1130 may be used to analyze a plurality of inputs andproduce a plurality of outputs corresponding to the inputs. Further, insome examples, information based on the plurality of outputs may beprovided to at least one user. Further, in some examples, one or morefeedbacks related to the information based on the plurality of outputsand provided to the at least one user may be received from the at leastone user. Further, in some examples, the one or more feedbacks relatedto the information based on the plurality of outputs may be used toevaluate the updated version of the first inference model and obtain theevaluation result. For example, the inputs may comprise images, thecorresponding outputs may comprise segmentation results of the images,the images and the resulting segments may be presented to the at leastone user, the feedbacks may include an indication of the correctness ofthe segmentation, and the evaluation result may be based on theindication of the correctness of the segmentation. In another example,the inputs may comprise images, the corresponding outputs may compriseclassification of objects depicted in the images, the images and theproduced classifications may be presented to the at least one user, thefeedbacks may include an indication of the correctness of the producedclassifications, and the evaluation result may be based on theindication of the correctness of the produced classifications. In yetanother example, the inputs may comprise images, the correspondingoutputs may comprise localization of objects in the images, the imagesand the estimated locations may be presented to the at least one user,the feedbacks may include an indication of the correctness of theestimated locations, and the evaluation result may be based on theindication of the correctness of the estimated locations. In anadditional example, the inputs may comprise audio segments, thecorresponding outputs may comprise identification of speakers in theaudio segments, the audio segments and/or the identification of speakersmay be presented to the at least one user, the feedbacks may include anindication of the correctness of the identification of speakers, and theevaluation result may be based on the indication of the correctness ofthe identification of speakers. In yet another example, the inputs maycomprise textual information, the corresponding outputs may compriseestimated properties of the textual information, the textual informationand/or the estimated properties of the textual information may bepresented to the at least one user, the feedbacks may include anindication of the correctness of the estimated properties of the textualinformation, and the evaluation result may be based on the indication ofthe correctness of the estimated properties of the textual information.

In some examples, the first inference model obtained by Step 1110 may beconfigured to analyze visual data (for example visual data captured froman environment), and the second inference model obtained by Step 1110may be configured to analyze audio data (for example audio data capturedin conjunction with the visual data). For example, training examples ofthe first version of a set of training examples of Step 1110 and/or ofthe updated version of the set of training examples of Step 1120 maycomprise video data comprising both image frames and corresponding audio(for example video data captured from an environment), the firstinference model obtained by Step 1110 and/or the updated version of thefirst inference model generated by Step 1130 may be trained using atleast part of the image frames and configured to analyze images, and thesecond inference model obtained by Step 1110 and/or an updated versionof the second inference model may be trained using at least part of theaudio data and configured to analyze audio. In another example, trainingexamples of the first version of a set of training examples of Step 1110and/or of the updated version of the set of training examples of Step1120 may comprise image data and corresponding audio data related to theimage data (for example, audio data captured in conjunction with theimage data, audio data comprising a description of the image data, andso forth), the first inference model obtained by Step 1110 and/or theupdated version of the first inference model generated by Step 1130 maybe trained using at least part of the image data and configured toanalyze images, and the second inference model obtained by Step 1110and/or an updated version of the second inference model may be trainedusing at least part of the audio data and configured to analyze audio.

In some examples, the first inference model obtained by Step 1110 may beconfigured to analyze audio data (for example audio data captured froman environment), and the second inference model obtained by Step 1110may be configured to analyze visual data (for example visual datacaptured in conjunction with the audio data). For example, trainingexamples of the first version of a set of training examples of Step 1110and/or of the updated version of the set of training examples of Step1120 may comprise video data comprising both image frames andcorresponding audio (for example video data captured from anenvironment), the second inference model obtained by Step 1110 and/or anupdated version of the second inference model may be trained using atleast part of the image frames and configured to analyze images, and thefirst inference model obtained by Step 1110 and/or the updated versionof the first inference model generated by Step 1130 may be trained usingat least part of the audio data and configured to analyze audio. Inanother example, training examples of the first version of a set oftraining examples of Step 1110 and/or of the updated version of the setof training examples of Step 1120 may comprise image data andcorresponding audio data related to the image data (for example, audiodata captured in conjunction with the image data, audio data comprisinga description of the image data, and so forth), the second inferencemodel obtained by Step 1110 and/or an updated version of the secondinference model may be trained using at least part of the image data andconfigured to analyze images, and the first inference model obtained byStep 1110 and/or the updated version of the first inference modelgenerated by Step 1130 may be trained using at least part of the audiodata and configured to analyze audio.

In some examples, the first inference model obtained by Step 1110 may beconfigured to analyze audio data (for example audio data captured froman environment), and the second inference model obtained by Step 1110may be configured to analyze textual information associated with theaudio data. For example, the audio data may be analyzed by the firstinference model to generate the textual information (for example,transcribed, summarized, analyzed to identify keywords, and so forth),and the second inference model may analyze the textual information togenerate additional information (for example, the second inference modelmay include natural language processing model, may analyze the textualinformation to identify topics, may analyze the textual information toidentify conversations, may analyze the textual information to identifysentiment, and so forth). In another example, the textual informationmay include contextual information related to the audio data (such asnames of speakers, name of location in which the audio data wascaptured, agenda and/or a written summary of a meeting that the audiocaptured from, and so forth), the first inference model may be used toanalyze the audio data to obtain first information (such as sentiment,speech prosody, loudness, etc.), the second inference model may be usedto analyze the textual information to obtain context (name of speaker,location, topic, etc.), and in some examples the first information andthe context may be analyzed to identify inter-correlations.

In some examples, the first inference model obtained by Step 1110 may beconfigured to analyze textual information associated with audio data,and the second inference model obtained by Step 1110 may be configuredto analyze the associated audio data. For example, the audio data may beanalyzed by the second inference model to generate the textualinformation (for example, transcribed, summarized, analyzed to identifykeywords, and so forth), and the first inference model may analyze thetextual information to generate additional information (for example, thefirst inference model may include natural language processing model, mayanalyze the textual information to identify topics, may analyze thetextual information to identify conversations, may analyze the textualinformation to identify sentiment, and so forth). In another example,the textual information may include contextual information related tothe audio data (such as names of speakers, name of location in which theaudio data was captured, agenda and/or a written summary of a meetingthat the audio captured from, and so forth), the first inference modelmay be used to analyze the textual information to obtain context (nameof speaker, location, topic, etc.), the second inference model may beused to analyze the audio data to obtain first information (such assentiment, speech prosody, loudness, etc.), and in some examples thefirst information and the context may be analyzed to identifyinter-correlations.

In some examples, the first inference model obtained by Step 1110 may beconfigured to analyze image data (for example image data captured froman environment), and the second inference model obtained by Step 1110may be configured to analyze textual information (for example textualinformation associated with the image data). For example, the image datamay be analyzed by the first inference model to generate the textualinformation (for example, extracting text depicted in the image data,textually describing objects detected in the image data, textuallydescribing a scene depicted in the image data, and so forth), and thesecond inference model may analyze the generated textual information togenerate additional information (for example, the second inference modelmay include natural language processing model, may analyze the textualinformation to identify contextual information, and so forth). Inanother example, the textual information may include contextualinformation related to the image data (such as names of objects depictedin the image data, name of a location depicted in the image data, nameof a person that captured the image data, agenda and/or a writtensummary of a meeting depicted in the image data, and so forth), thefirst inference model may be used to analyze the image data to obtainfirst information (such as text depicted in the image data, textualdescription of the image data, etc.), the second inference model may beused to analyze the textual information to obtain context, and in someexamples the first information and the context may be analyzed toidentify inter-correlations.

In some examples, the first inference model obtained by Step 1110 may beconfigured to analyze textual information (for example textualinformation associated with image data), and the second inference modelobtained by Step 1110 may be configured to analyze image data (forexample image data associated with the textual information). Forexample, the image data may be analyzed by the second inference model togenerate the textual information (for example, extracting text depictedin the image data, textually describing objects detected in the imagedata, textually describing a scene depicted in the image data, and soforth), and the first inference model may analyze the generated textualinformation to generate additional information (for example, the secondinference model may include natural language processing model, mayanalyze the textual information to identify contextual information, andso forth). In another example, the textual information may includecontextual information related to the image data (such as names ofobjects depicted in the image data, name of a location depicted in theimage data, name of a person that captured the image data, agenda and/ora written summary of a meeting depicted in the image data, and soforth), the second inference model may be used to analyze the image datato obtain first information (such as text depicted in the image data,textual description of the image data, etc.), the first inference modelmay be used to analyze the textual information to obtain context, and insome examples the first information and the context may be analyzed toidentify inter-correlations.

In some examples, the first inference model obtained by Step 1110 may beconfigured to detect a first type of items in image data captured froman environment, and the second inference model obtained by Step 1110 maybe configured to determine properties of the detected items by analyzingthe image data. For example, the first type of items may include faces,the first inference model may include a face detector, and the secondinference model may be configured to estimate properties of the detectedfaces, such as age, gender, hair color, and so forth. In anotherexample, the first type of items may include objects of selected types,the first inference model may include an object detector, and the secondinference model may be configured to estimate properties of the detectedobjects, such as size, volume, color, and so forth.

FIG. 12 is a schematic illustration of example an environment 1200consistent with an embodiment of the present disclosure. In thisexample, environment 1200 may comprise device 1202A and device 1202B.Further, in this example, device 1202A may comprise at least sensors1204A and inference models 1206A, and device 1202B may comprise at leastsensors 1204B and inference models 1206B. Some examples of such devicesmay include personalized devices (such as smartphone, tablets, PCs,etc.), computing devices (such as apparatus 200, server 300, cloudplatform 400, etc.), and so forth. Some examples of sensors 1204A and1204B may include any of the sensors described in the presentdisclosure. Some examples of inference models 1206A and 1206B mayinclude any of the inference models described in the present disclosure.In this example, environment 1200 may further comprise road 1210 andvehicles 1220, 1222 and 1224. In some examples, device 1202A may useinference models 1206A to analyze data captured using sensors 1204A, anddevice 1202B may use inference models 1206B to analyze data capturedusing sensors 1204B. In some examples, both device 1202A and device1202B may use their sensors to capture data from the same portion ofenvironment 1200 and use their inference models to analyze the captureddata and monitor the same portion of environment 1200. For example, bothdevices may monitor the same vehicle, such as vehicle 1222. In someexamples, device 1202A may use sensors 1204A to capture data from aportion of environment 1200 that is not monitored by sensors 1204Band/or may use inference models 1206A to analyze data captured from aportion of environment 1200 that is not monitored by inference models1206B. For example, device 1202A may monitor vehicle 1224 that is notmonitored by device 1202B. In some examples, method 1100 may be used toselectively update inference models 1206A and/or inference models 1206B.For example, the first inference model and the second inference model ofStep 1110 may be included in inference models 1206A, the updated versionof the set of training examples of Step 1120 may include trainingexamples compatible with sensors 1204B, in Step 1130 an updated versionof the first inference model may be generated based on the updatedversion of the set of training examples, the updated version of thefirst inference model may be included in inference models 1206B and/orused to analyze data captured by sensors 1204B, the updated version ofthe first inference model may be evaluated by Step 1140, and in Step1150 the evaluation result may be used to determine whether to updatethe second inference model and/or whether to include the updated versionof the second inference model in inference models 1206B and/or whetherto use the updated version of the second inference model to analyze datacaptured by sensors 1204B. In another example, the first inference modelof Step 1110 may be included in inference models 1206A and the secondinference model of Step 1110 may be included in inference models 1206B,in Step 1130 an updated version of the first inference model may begenerated based on the updated version of the set of training examples,the updated version of the first inference model may be included ininference models 1206A and/or used to analyze data captured by sensors1204A, the updated version of the first inference model may be evaluatedby Step 1140, and in Step 1150 the evaluation result may be used todetermine whether to update the second inference model and/or whether toinclude the updated version of the second inference model in inferencemodels 1206B and/or whether to use the updated version of the secondinference model to analyze data captured by sensors 1204B. In yetanother example, the first inference model and the second inferencemodel of Step 1110 may be included in inference models 1206A, in Step1130 an updated version of the first inference model may be generatedbased on the updated version of the set of training examples, theupdated version of the first inference model may be included ininference models 1206A and/or used to analyze data captured by sensors1204A, the updated version of the first inference model may be evaluatedby Step 1140, and in Step 1150 the evaluation result may be used todetermine whether to update the second inference model and/or whether toinclude the updated version of the second inference model in inferencemodels 1206A and/or whether to use the updated version of the secondinference model to analyze data captured by sensors 1204A.

FIG. 13 illustrates an example of a method 1300 for selecting hyperparameters for machine learning algorithms based on past trainingresults. In this example, method 1300 may comprise: obtaining differentgroups of values of hyper-parameters (Step 1310); obtaining results oftraining a machine learning algorithm using different training setsand/or the different groups of values of hyper-parameters (Step 1320);and using the results and the corresponding groups of values ofhyper-parameters to select values of hyper-parameters for a prospectivetraining of the machine learning algorithm (Step 1330). In someimplementations, method 1300 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Insome implementations, one or more steps illustrated in FIG. 13 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, in Step 1310 different groups of values of hyperparameters of a machine learning algorithm may be obtained. For example,in Step 1310 a first group of values of hyper parameters, a second groupof values of hyper parameters and a third group of values of hyperparameters may be obtained, where the second group of values of hyperparameters may be identical to or different from the first group ofvalues of hyper parameters, and the third group of values of hyperparameters may be identical to or different from the first group ofvalues of hyper parameters and the second group of values of hyperparameters. For example, at least part of the groups of values of hyperparameters may be may be read from memory (for example from memory unit210, shared memory module 410, etc.), read from a blockchain, receivedthrough a communication network (such as communication network 130)using a communication device (such as communication module 230),received from an external device (such as mobile phone 111, tablet 112,PC 113, remote storage 140, NAS 150, server 300, cloud platform 400,etc.), generated (for example using another instance of method 1300),selected (for example of a plurality of alternative groups of values ofhyper parameters), and so forth. Some examples of such hyper parametersmay include optimization criterion, optimization method, hyperparameters of the optimization method, stopping condition, initialguess, and so forth. In one example, the machine learning algorithm mayinclude an artificial neural network (such as a deep neural network, aconvolutional neural network, a recursive neural network, etc.) and someexamples of such hyper parameters may include type of artificial neuralnetwork, number of layers, types of layers, number of artificial neurons(for example, in a layer), types of artificial neurons, types ofactivation functions, hyper parameters of activation functions, learningrate, learning rate update method, hyper parameters of learning rateupdate method, batch size, initialization method, hyper parameters ofthe initialization method, dropout rate, and so forth. In anotherexample, the machine learning algorithm may include a random forest, andsome additional examples of hyper parameters may include maximal depth,number of trees, hyper parameters for controlling splits, and so forth.In yet another example, the machine learning algorithm may include asupport vector machine, and some additional examples of hyper parametersmay include penalty parameter, kernel type, hyper parameters of thekernel, and so forth. In an additional example, the machine learningalgorithm may include a classification algorithm, and some additionalexamples of hyper parameters may include number of classes, selection ofclasses of a plurality of alternative classes, weights for classes,weights for training examples, normalization method, and so forth. Inyet another example, the machine learning algorithm may include aregression algorithm, and some additional examples of hyper parametersmay include error function, hyper parameters of the error function, andso forth. In an additional example, the machine learning algorithm mayinclude a clustering algorithm (and/or a segmentation algorithm), andsome additional examples of hyper parameters may include number ofdesired clusters (and/or number of desired segments), minimal clustersize (and/or minimal segment size), maximal cluster size (and/or maximalsegment size), and so forth. In yet another example, the machinelearning algorithm may include an ensemble algorithm, and someadditional examples of hyper parameters may include number of basemodels, hyper parameters for one or more of the base models, and soforth.

In some embodiments, in Step 1320 results of training a machine learningalgorithm using one or more pluralities of training examples and one ormore groups of values of hyper parameters may be obtained. In someexamples, in Step 1320 results of training the machine learningalgorithm using a plurality of training examples and the differentgroups of values of hyper parameters obtained by Step 1310 may beobtained. In some example, in Step 1320 results of training the machinelearning algorithm using different pluralities of training examples anda group of values of hyper parameters may be obtained. For example, inStep 1320 a first result of training the machine learning algorithmusing a first plurality of training examples and the first group ofvalues of hyper parameters obtained by Step 1310 may be obtained, asecond result of training the machine learning algorithm using a secondplurality of training examples and the second group of values of hyperparameters obtained by Step 1310 may be obtained, a third result oftraining the machine learning algorithm using a third plurality oftraining examples and the third group of values of hyper parametersobtained by Step 1310 may be obtained, and so forth. In another example,in Step 1320 a first result of training the machine learning algorithmusing a first plurality of training examples and the first group ofvalues of hyper parameters obtained by Step 1310 may be obtained, asecond result of training the machine learning algorithm using the firstplurality of training examples and the second group of values of hyperparameters obtained by Step 1310 may be obtained, a third result oftraining the machine learning algorithm using the first plurality oftraining examples and the third group of values of hyper parametersobtained by Step 1310 may be obtained, and so forth. For example, inStep 1320 the results of training a machine learning algorithm using theone or more pluralities of training examples and/or the one or moregroups of values of hyper parameters may be read from memory (forexample from memory unit 210, shared memory module 410, etc.), read froma blockchain, received through a communication network (such ascommunication network 130) using a communication device (such ascommunication module 230), received from an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.), generated (for example bytraining the machine learning algorithm using the one or morepluralities of training examples and/or the one or more groups of valuesof hyper parameters as described above, by using an inference modelgenerated by training the machine learning algorithm using the one ormore pluralities of training examples and/or the one or more groups ofvalues of hyper parameters as described above to generate outputscorresponding to example inputs, by generating statistics using theseoutputs, etc.), selected (for example using a data structure holding aplurality of alternative results corresponding to one or morepluralities of training examples and/or to one or more groups of valuesof hyper parameters), and so forth. For example, the results of traininga machine learning algorithm using the one or more pluralities oftraining examples and/or the one or more groups of values of hyperparameters may comprise an inference model that is a result of trainingthe machine learning algorithm, such as a predictive model, aclassification model, a regression model, a clustering model, asegmentation model, an artificial neural network (such as a deep neuralnetwork, a convolutional neural network, a recursive neural network,etc.), a random forest, a support vector machine, and so forth. Inanother example, the results of training a machine learning algorithmusing the one or more pluralities of training examples and/or the one ormore groups of values of hyper parameters may comprise outputs generatedusing example inputs and such inference model. In yet another example,the results of training a machine learning algorithm using the one ormore pluralities of training examples and/or the one or more groups ofvalues of hyper parameters may comprise information about a comparisonof such outputs with desired outputs, such as numbers and/or ratios offalse positives and/or false negatives and/or true positives and/or truenegatives, accuracy, precision, F score, confusion matrix, otherstatistics about classification errors, sum of absolute errors, sum ofsquare errors, other statistics about regression errors, intersectionover union scores, segmentation error scores, clustering error scores,and so forth.

In some embodiments, in Step 1330 the results obtained by Step 1320and/or the corresponding groups of values of hyper parameters obtainedby Step 1310 may be used to select at least one value of a hyperparameter for a prospective training of a machine learning algorithm(for example, for a prospective training of the machine learningalgorithm of Step 1320, for a prospective training of a differentmachine learning algorithm, and so forth). For example, in Step 1330 thefirst result, the second result and the third result obtained by Step1320, and the first group of values of hyper parameters, the secondgroup of values of hyper parameters and the third group of values ofhyper parameters obtained by Step 1310 may be used to select at leastone value of a hyper parameter for a prospective training of the machinelearning algorithm. For example, a machine learning model may be trainedusing training examples to select hyper parameters for prospectivetrainings of machine learning algorithms based on results of trainingmachine learning algorithms and corresponding groups of values of hyperparameters, and the trained machine learning algorithm may be used toanalyze the results obtained by Step 1320 and the corresponding groupsof values of hyper parameters obtained by Step 1310 to select the atleast one value of the hyper parameter for the prospective training of amachine learning algorithm. Such training example may include aplurality of results of training machine learning algorithms with thecorresponding groups of values of hyper parameters, together with adesired selection of at least one value of the hyper parameter for theprospective training of the machine learning algorithm. In anotherexample, an artificial neural network may be configured to select hyperparameters for prospective trainings of machine learning algorithmsbased on results of training machine learning algorithms andcorresponding groups of values of hyper parameters, and the artificialneural network may be used to analyze the results obtained by Step 1320and the corresponding groups of values of hyper parameters obtained byStep 1310 to select the at least one value of the hyper parameter forthe prospective training of a machine learning algorithm. In yet anotherexample, a weight may be assigned to groups of values of hyperparameters obtained by Step 1310 according to results obtained by Step1320, a weighted average of the values of hyper parameters may becalculated using the assigned weights, and the selection of the at leastone value of the hyper parameter for the prospective training of amachine learning algorithm may be based on the calculated weightedaverage. In an additional example, a weight may be assigned to groups ofvalues of hyper parameters obtained by Step 1310 according to resultsobtained by Step 1320, weighted voting among the values of hyperparameters may be calculated using the assigned weights, and the atleast one value of the hyper parameter for the prospective training of amachine learning algorithm may be selected to be similar or identical tothe values of hyper parameters selected through voting. For example, thehigher weights may be assigned to values of hyper parameterscorresponding to better results, higher weights may be assigned tovalues of hyper parameters corresponding to lesser results, and soforth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first learning rate value, the second groupof values of hyper parameters obtained by Step 1310 may comprise asecond learning rate value, the third group of values of hyperparameters obtained by Step 1310 may comprise a third learning ratevalue. Further, in some examples, in Step 1330 the first result, thesecond result and the third result obtained by Step 1320, and the firstlearning rate value, the second learning rate value and the thirdlearning rate value obtained by Step 1310 may be used to select alearning rate value for the prospective training of the machine learningalgorithm, for example as described above using a trained machinelearning model, using an artificial neural network, using weightedaverage of the learning rate values, using weighted voting of thelearning rate values, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of a learning rate updatemethod, the second group of values of hyper parameters obtained by Step1310 may comprise a second selection of a learning rate update method,the third group of values of hyper parameters obtained by Step 1310 maycomprise a third selection of a learning rate update method. Further, insome examples, in Step 1330 the first result, the second result and thethird result obtained by Step 1320, and the first selection of alearning rate update method, the second selection of a learning rateupdate method and the third selection of a learning rate update methodobtained by Step 1310 may be used to select a learning rate updatemethod for the prospective training of the machine learning algorithm,for example as described above using a trained machine learning model,using an artificial neural network, using weighted voting of thelearning rate update methods, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first batch size, the second group of valuesof hyper parameters obtained by Step 1310 may comprise a second batchsize, the third group of values of hyper parameters obtained by Step1310 may comprise a third batch size. Further, in some examples, in Step1330 the first result, the second result and the third result obtainedby Step 1320, and the first batch size, the second batch size and thethird batch size obtained by Step 1310 may be used to select a batchsize for the prospective training of the machine learning algorithm, forexample as described above using a trained machine learning model, usingan artificial neural network, using weighted average of the batch sizes,using weighted voting of the batch sizes, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first network structure, the second group ofvalues of hyper parameters obtained by Step 1310 may comprise a secondnetwork structure, the third group of values of hyper parametersobtained by Step 1310 may comprise a third network structure. Further,in some examples, in Step 1330 the first result, the second result andthe third result obtained by Step 1320, and the first network structure,the second network structure and the third network structure obtained byStep 1310 may be used to select a network structure for the prospectivetraining of the machine learning algorithm, for example as describedabove using a trained machine learning model, using an artificial neuralnetwork, using weighted voting of the network structure, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first number of layers, the second group ofvalues of hyper parameters obtained by Step 1310 may comprise a secondnumber of layers, the third group of values of hyper parameters obtainedby Step 1310 may comprise a third number of layers. Further, in someexamples, in Step 1330 the first result, the second result and the thirdresult obtained by Step 1320, and the first number of layers, the secondnumber of layers and the third number of layers obtained by Step 1310may be used to select a number of layers for the prospective training ofthe machine learning algorithm, for example as described above using atrained machine learning model, using an artificial neural network,using weighted average of the numbers of layers, using weighted votingof the numbers of layers, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first number of artificial neurons, thesecond group of values of hyper parameters obtained by Step 1310 maycomprise a second number of artificial neurons, the third group ofvalues of hyper parameters obtained by Step 1310 may comprise a thirdnumber of artificial neurons. Further, in some examples, in Step 1330the first result, the second result and the third result obtained byStep 1320, and the first number of artificial neurons, the second numberof artificial neurons and the third number of artificial neuronsobtained by Step 1310 may be used to select a number of artificialneurons for the prospective training of the machine learning algorithm,for example as described above using a trained machine learning model,using an artificial neural network, using weighted average of thenumbers of artificial neurons, using weighted voting of the numbers ofartificial neurons, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of an activation function,the second group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of an activation function, the third groupof values of hyper parameters obtained by Step 1310 may comprise a thirdselection of an activation function. Further, in some examples, in Step1330 the first result, the second result and the third result obtainedby Step 1320, and the first selection of an activation function, thesecond selection of an activation function and the third selection of anactivation function obtained by Step 1310 may be used to select anactivation function for the prospective training of the machine learningalgorithm, for example as described above using a trained machinelearning model, using an artificial neural network, using weightedvoting of the activation functions, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of a loss function, thesecond group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of a loss function, the third group ofvalues of hyper parameters obtained by Step 1310 may comprise a thirdselection of a loss function. Further, in some examples, in Step 1330the first result, the second result and the third result obtained byStep 1320, and the first selection of a loss function, the secondselection of a loss function and the third selection of a loss functionobtained by Step 1310 may be used to select a loss function for theprospective training of the machine learning algorithm, for example asdescribed above using a trained machine learning model, using anartificial neural network, using weighted voting of the loss functions,and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of an initialization method,the second group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of an initialization method, the third groupof values of hyper parameters obtained by Step 1310 may comprise a thirdselection of an initialization method. Further, in some examples, inStep 1330 the first result, the second result and the third resultobtained by Step 1320, and the first selection of an initializationmethod, the second selection of an initialization method and the thirdselection of an initialization method obtained by Step 1310 may be usedto select an initialization method for the prospective training of themachine learning algorithm, for example as described above using atrained machine learning model, using an artificial neural network,using weighted voting of the initialization methods, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of a kernel function, thesecond group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of a kernel function, the third group ofvalues of hyper parameters obtained by Step 1310 may comprise a thirdselection of a kernel function. Further, in some examples, in Step 1330the first result, the second result and the third result obtained byStep 1320, and the first selection of a kernel function, the secondselection of a kernel function and the third selection of a kernelfunction obtained by Step 1310 may be used to select a kernel functionfor the prospective training of the machine learning algorithm, forexample as described above using a trained machine learning model, usingan artificial neural network, using weighted voting of the kernelfunctions, and so forth. In some examples, the first selection of akernel function may include a selection of a polynomial kernel with afirst degree, the second selection of a kernel function may include aselection of a polynomial kernel with a second degree (in some examplesthe second degree may be identical to the first degree while in otherexamples the second degree may be different from the first degree), andthe third selection of a kernel function may include a selection of aRadial Basis Function (RBF) kernel with a first free parameter. Further,in one example, in response to a first group of results obtained by Step1320, an RBF kernel may be selected for the prospective training of themachine learning algorithm, while in response to a second group ofresults obtained by Step 1320, a polynomial kernel may be selected forthe prospective training of the machine learning algorithm. In anotherexample, in response to a first group of results obtained by Step 1320,a polynomial kernel with a degree closer to the first degree than to thesecond degree may be selected for the prospective training of themachine learning algorithm, while in response to a second group ofresults obtained by Step 1320, a polynomial kernel with a degree closerto the second degree than to the first degree may be selected for theprospective training of the machine learning algorithm. In yet anotherexample, in response to a first group of results obtained by Step 1320,an RBF kernel with a free parameter identical to the first freeparameter may be selected for the prospective training of the machinelearning algorithm, while in response to a second group of resultsobtained by Step 1320, an RBF kernel with a free parameter differentfrom the first free parameter may be selected for the prospectivetraining of the machine learning algorithm.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of a similarity function,the second group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of a similarity function, the third group ofvalues of hyper parameters obtained by Step 1310 may comprise a thirdselection of a similarity function. Further, in some examples, in Step1330 the first result, the second result and the third result obtainedby Step 1320, and the first selection of a similarity function, thesecond selection of a similarity function and the third selection of asimilarity function obtained by Step 1310 may be used to select asimilarity function for the prospective training of the machine learningalgorithm, for example as described above using a trained machinelearning model, using an artificial neural network, using weightedvoting of the similarity functions, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of a distance measure, thesecond group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of a distance measure, the third group ofvalues of hyper parameters obtained by Step 1310 may comprise a thirdselection of a distance measure. Further, in some examples, in Step 1330the first result, the second result and the third result obtained byStep 1320, the first selection of a distance measure, the secondselection of a distance measure and the third selection of a distancemeasure obtained by Step 1310 may be used to select a distance measurefor the prospective training of the machine learning algorithm, forexample as described above using a trained machine learning model, usingan artificial neural network, using weighted voting of the distancemeasures, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first selection of a stopping condition, thesecond group of values of hyper parameters obtained by Step 1310 maycomprise a second selection of a stopping condition, the third group ofvalues of hyper parameters obtained by Step 1310 may comprise a thirdselection of a stopping condition. Further, in some examples, in Step1330 the first result, the second result and the third result obtainedby Step 1320, and the first selection of a stopping condition, thesecond selection of a stopping condition and the third selection of astopping condition obtained by Step 1310 may be used to select astopping condition for the prospective training of the machine learningalgorithm, for example as described above using a trained machinelearning model, using an artificial neural network, using weightedvoting of the stopping conditions, and so forth.

In some examples, the first group of values of hyper parameters obtainedby Step 1310 may comprise a first sampling criterion used to sample thefirst plurality of training examples from a base group of trainingexamples, the second group of values of hyper parameters obtained byStep 1310 may comprise a second sampling criterion used to sample thesecond plurality of training examples from the base group of trainingexamples, the third group of values of hyper parameters obtained by Step1310 may comprise a third sampling criterion used to sample the thirdplurality of training examples from a base group of training examples.Further, in some examples, in Step 1330 the first result, the secondresult and the third result obtained by Step 1320, and the firstsampling criterion, the second sampling criterion and the third samplingcriterion obtained by Step 1310 may be used to select a samplingcriterion for sampling a plurality of training examples from the basegroup of training examples for the prospective training of the machinelearning algorithm, for example as described above using a trainedmachine learning model, using an artificial neural network, usingweighted voting of the sampling criteria, and so forth. Some examples ofsuch sampling criteria may include desired ratios among different typesof samples (for example among examples of different classes, of samplesof different properties, etc.), desired amount of different types ofsamples (for example of different classes, of samples of differentproperties, etc.), sampling selection rules, and so forth. In someexamples, the first sampling criterion may specify a first set ofdesired ratios among the different types of samples, the second samplingcriterion may specify a second set of desired ratios among the differenttypes of samples, and the third sampling criterion may specify a thirdset of desired ratios among the different types of samples. Further, inone example, in response to a first group of results obtained by Step1320, a set of desired ratios among the different types of samplescloser to the first set of ratios than to the second set may be selectedfor the prospective training of the machine learning algorithm, while inresponse to a second group of results obtained by Step 1320, a set ofdesired ratios among the different types of samples closer to the secondset of ratios than to the first set may be selected for the prospectivetraining of the machine learning algorithm. In another example, inresponse to a first value of the third result obtained by Step 1320, aset of desired ratios among the different types of samples closer to thefirst set of ratios than to the second set may be selected for theprospective training of the machine learning algorithm, while inresponse to a second value of the third result obtained by Step 1320, aset of desired ratios among the different types of samples closer to thesecond set of ratios than to the first set may be selected for theprospective training of the machine learning algorithm.

In some embodiments, the machine learning algorithm of Step 1320 may betrained using the first plurality of training examples of Step 1320 andthe first group of values of hyper parameters obtained by Step 1310 togenerate a first inference model, for example as described above.Further, in some examples, the first inference model and a plurality oftesting examples may be used to generate a first plurality of outputscorresponding to the testing examples. Further, in some examples, thefirst plurality of outputs and a plurality of desired results may becompared to determine the first result obtained by Step 1320. Forexample, the first result obtained by Step 1320 may be based on afunction of one or more of numbers and/or ratios of false positivesand/or false negatives and/or true positives and/or true negatives,accuracy, precision, F score, confusion matrix, other statistics aboutclassification errors, sum of absolute errors, sum of square errors,other statistics about regression errors, intersection over unionscores, segmentation error scores, clustering error scores, and soforth.

In some embodiments, an embedding in a mathematical space of at least avalue of the first group of values of hyper parameters, a value of thesecond group of values of hyper parameters and a value of the thirdgroup of values of hyper parameters may be determined. For example, theembedding in the mathematical space may be determined based on the valueof the first group of values of hyper parameters, the value of thesecond group of values of hyper parameters and the value of the thirdgroup of values of hyper parameters. In another example, the embeddingin the mathematical space may be determined based on the first result,the second result and the third result obtained by Step 1320. In yetanother example, the embedding in the mathematical space may bedetermined using Principal Component Analysis procedure. In anadditional example, the embedding in the mathematical space may bedetermined using an artificial neural network. In yet another example,the embedding in the mathematical space may be determined using atransformation function. Further, in some examples, in Step 1330 thedetermined embedding of the values in the mathematical space may be usedto select the at least one value of the hyper parameter for theprospective training of the machine learning algorithm. For example,each point in the mathematical space may correspond to a value of thehyper parameter, a centroid of the embedded values in the mathematicalspace may be calculated, and the at least one value of the hyperparameter for the prospective training of the machine learning algorithmmay be selected based on the value of the hyper parameter correspondingto the centroid. In another example, the embedding of the values in themathematical space may be analyzed to select one of the embedded values,and the group of values of hyper parameters corresponding to theselected embedded value may be selected for the prospective training ofthe machine learning algorithm. For example, the embedded value with themaximal distance to the nearest other embedded value may be select. Inanother example, the embedded value with the minimal average distance tothe k nearest other embedded values may be selected.

In some examples, it may be determined whether the first result ispreferable to the second result and the third result. For example, eachresult may correspond to a score (such as a function of one or more ofnumber of true positives, ratio of true positives, number of truenegatives, ratio of true negatives, accuracy, precision, recall, Fscore, confusion matrix, number of false positives, ratio of falsepositives, number of false negatives, ratio of false negatives,statistics about classification errors, sum of absolute errors, sum ofsquare errors, statistics about regression errors, intersection overunion scores, segmentation error scores, clustering error scores, and soforth), and the result corresponding to the higher score may bedetermined to be preferable. Further, in some examples, in response tothe determination that the first result is preferable to the secondresult and the third result, the at least one value of the hyperparameter for the prospective training of the machine learning algorithmmay be selected to be closer to the first group of values of hyperparameters than to the second group of values of hyper parameters and tothe third group of values of hyper parameters according to the embeddingin the mathematical space, and in response to the determination that thefirst result is not preferable to the second result and the thirdresult, the at least one value of the hyper parameter for theprospective training of the machine learning algorithm may be selectedto be farther from the first group of values of hyper parameters thanfrom at least one of the second group of values of hyper parameters andthe third group of values of hyper parameters according to the embeddingin the mathematical space.

In some embodiments, the machine learning algorithm of Step 1320 may betrained using the first plurality of training examples of Step 1320 andthe first group of values of hyper parameters obtained by Step 1310 togenerate a first inference model, for example as described above.Further, in some examples, the first inference model and a plurality oftesting examples may be used to generate a first plurality of outputscorresponding to the testing examples. Further, in some examples,information based on the first plurality of outputs may be provided toat least one user. For example, the information may be providedvisually, may be provided audibly, may be provided textually, may beprovided through a personalized device associated with the user, may beprovided through a public device, and so forth. Further, in someexamples, one or more feedbacks related to the information based on thefirst plurality of outputs may be received from the at least one user.Further, in some examples, the one or more feedbacks related to theinformation based on the first plurality of outputs may be used todetermine the first result. Some examples of possible implementations ofa user interface for providing information based on at least one outputto a user and receiving feedbacks related to the information areprovided in FIGS. 14A and 14B as described below, and the such userinterface may be used repeatedly to obtain feedbacks related toinformation based on the first plurality of outputs. For example, inresponse to a first feedback related to the information based on thefirst plurality of outputs, a first value of the first result may bedetermined, and in response to a second feedback related to theinformation based on the first plurality of outputs, a second value ofthe first result may be determined.

FIG. 14A is a schematic illustration of a user interface 1400 consistentwith an embodiment of the present disclosure. In this example, userinterface 1400 may comprise a presentation of input image 1402, apresentation of output image 1404, a presentation of queries 1406 to theuser requesting the user to grade output image 1404. In this example,input image 1402 may comprise a depiction of a person in an environmentof a yard, output image 1404 may comprise a depiction of the yardwithout the person generated by removing the person from image 1402 andinpainting the resulting image, for example automatically using one ormore inference models. In this example, queries 1406 may request theuser to indicate whether the entire object was removed, to indicatewhether no parts of other objects were removed, to indicate the qualityof the inpainting, and so forth.

FIG. 14B is a schematic illustration of a user interface 1420 consistentwith an embodiment of the present disclosure. In this example, userinterface 1420 may comprise a presentation of input image 1422, apresentation of an overlay 1424 over input image 1422 providing outputinformation, a presentation of queries 1426 to the user requesting theuser to grade the provided output information. In this example, inputimage 1422 may comprise a depiction of a person in an environment of ayard, overlay 1424 may comprise a bounding box identifying a determinedlocation of a detect item accompanied with a visual presentation of atextual description of the determined type of the detected item (in thisexample, “Person”), for example as determined automatically using one ormore inference models. In this example, queries 1426 may request theuser to indicate whether the determined class of the item is correct, toindicate whether the determined location of the item is correct, and soforth.

FIG. 15 illustrates an example of a method 1500 for class aware objectmarking. In this example, method 1500 may comprise: receiving anindication of a selected object type (Step 1510); presenting an image toa user (Step 1520); receiving an input to an object selection tool (Step1530); and selecting a region of the image corresponding to an object ofthe selected object type in the image based on the selected object typeand the input to the object selection tool (Step 1540). In someimplementations, method 1500 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Forexample, in some cases Step 1520 may be excluded from method 1500. Insome implementations, one or more steps illustrated in FIG. 15 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, in Step 1510 an indication of a selected objecttype of a plurality of alternative object types may be received. Somenon-limiting examples of such object types may include object togetherwith adjunct related objects, object without adjunct related objects,object together with occluded parts, object without occluded parts,single instance of an object of a specified type, one or more instancesof objects of a specified type, person, person together with objectsheld by the person, person not including objects held by the person,person including occluded parts, person except for occluded parts, male,female, face, face including occluded parts, face excluding occludedparts, face including hair, face excluding hair, vehicle, specific typesof vehicles (such as motorcycles, cars, trucks, bicycles, etc.), vehiclewith occluded parts, vehicle without occluded parts, motorcyclesincluding their rider, motorcycles not including their riders, and soforth. In some examples, the plurality of alternative object types maybe presented to the user, and in Step 1510 the indication of theselected object type of the plurality of alternative object types may bereceived from the user. In some examples, the image may be analyzed toselect the selected object type of the plurality of alternative objecttypes. For example, a machine learning model may be trained usingtraining examples to select object types based on images and/or videos,and the trained machine learning model may be used to analyze the imageand select the selected object type. Some examples of such trainingexample may include an image together with the desired selection of theobject type. In another example, an artificial neural network may beconfigured to select object types based on images and/or videos, and theartificial neural network may be used to analyze the image and selectthe selected object type. In some examples, the selected object type maybe selected of the plurality of alternative object types at random. Insome examples, the selected object type may be received from the userusing a user interface (for example, using a keyboard, using a computermouse, using a touch screen, using gestures, etc.), read from memory(for example from memory unit 210, shared memory module 410, etc.), readfrom a blockchain, received through a communication network (such ascommunication network 130) using a communication device (such ascommunication module 230), received from an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.), and so forth.

In some embodiments, in Step 1520 an image may be presented to a user.For example, an image may be presented to a user using a display screen,an augmented reality display system, a printer, and so forth. Forexample, the image may comprise a depiction of one or more items.

In some embodiments, in Step 1530 an input to an object selection toolmay be received, for example from the user. For example, the input tothe object selection tool may be received from the user using a userinterface (for example, using a keyboard, using a computer mouse, usinga touch screen, using gestures, etc.), read from memory (for examplefrom memory unit 210, shared memory module 410, etc.), read from ablockchain, received through a communication network (such ascommunication network 130) using a communication device (such ascommunication module 230), received from an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.), and so forth. Some non-limitingexamples of such inputs to the object selection tool may include anindication of one or more positions within the image presented by Step1520, an indication of one or more regions within the image presented byStep 1520, an indication of one or more contours within the imagepresented by Step 1520, an indication of one or more masks of the imagepresented by Step 1520, an indication of one or more subsets of thepixels of the image presented by Step 1520, and so forth.

In some embodiments, in Step 1540, a region of the image correspondingto an object of the selected object type of Step 1510 in the imagepresented by Step 1520 may be selected based on the input to the objectselection tool received by Step 1530 and the selected object type ofStep 1510. In some examples, in response to a first selected object typeand the input to the object selection tool, a first region of the imagecorresponding to an object of the first selected object type in theimage may be selected, and in response to a second selected object typeand the input to the object selection tool, a second region of the imagecorresponding to an object of the second selected object type in theimage may be selected, where the first region may differ from the secondregion. For example, a machine learning model may be trained usingtraining examples to select regions of images according to inputs toobject selection tools and object types, and the trained machinelearning model may be used to analyze the image and select the regionbased on the input to the object selection tool received by Step 1530and the selected object type of Step 1510. An example of such trainingexample may include an image with an input to an object selection tooland a selection of object type, together with the desired selection of aregion in the image. In another example, an artificial neural networkmay be configured to select regions of images according to inputs toobject selection tools and selection of object types, and the artificialneural network may be used to analyze the image and select the regionbased on the input to the object selection tool received by Step 1530and the selected object type of Step 1510. In yet another example, aplurality of alternative object selection tools may be available, whereeach one or the alternative object selection tools may correspond to aparticular object type, an object selection tool corresponding to theselected object type of Step 1510 may be selected of the plurality ofalternative object selection tools, the input to the object selectiontool received by Step 1530 may be provided to the selected objectselection tool corresponding to the selected object type of Step 1510,and selection of the region of the image by Step 1540 may be based onthe output of the selected object selection tool corresponding to theselected object type of Step 1510.

For example, the first selected object type of Step 1510 may be a persontogether objects held by the person and the second selected object typeof Step 1510 may be a person not including objects held by the person,in response to the first selected object type and the input to theobject selection tool received by Step 1530, a first region of the imagecorresponding to a person and an object held by the person in the imagemay be selected by Step 1540, and in response to the second selectedobject type and the input to the object selection tool received by Step1530, a second region of the image corresponding to the person withoutthe object held by the person in the image may be selected by Step 1540,and therefore the first region may differ from the second region. Someexamples of objects held by the person may include bags, cap,smartphone, jacket, and so forth. For example, Step 1520 may present animage of a person holding a bag in a room, and Step 1530 may receive abounding box around the person holding the bag, as illustrated in FIG.16A. Further, in this example, in response to an indication of aselected object type of a person together objects held by the personreceived by Step 1510 and the bounding box received by Step 1530, aregion of the image corresponding to the person and the bag held by theperson in the image may be selected by Step 1540 as illustrated in FIG.16B. Further, in this example, in response to an indication of aselected object type of a person not including objects held by theperson received by Step 1510 and the bounding box received by Step 1530,a region of the image corresponding to the person without the bag heldby the person in the image may be selected by Step 1540 as illustratedin FIG. 16C.

In another example, the first selected object type of Step 1510 may be aperson including occluded parts and the second selected object type ofStep 1510 may be a person except for occluded parts. in response to thefirst selected object type and the input to the object selection toolreceived by Step 1530, a first region of the image corresponding to aperson together with parts of the person that are occluded in the imagemay be selected by Step 1540, and in response to the second selectedobject type and the input to the object selection tool received by Step1530, a second region of the image corresponding to the person withoutthe parts of the person that are occluded in the image may be selectedby Step 1540, and therefore the first region may differ from the secondregion. In some examples, a machine learning model may be trained usingtraining examples to estimate the region of the image corresponding tothe person including to parts of the person that are occluded in theimage based on the visible parts of the person in the image and/or othervisual data in the image, and the trained machine learning model may beused to analyze the image and select the region of the image framecorresponding to the person together with parts of the person that areoccluded in the image. An example of such training example may includean image together with the desired selection of a region in the image.In some examples, the image may be an image frame from a video, andparts of the person that are occluded in the image frame may be visiblein other frames of the video (for example, in consecutive frames, in adifferent segment of the video, and so forth). Further, in one example,a machine learning model may be trained using training examples toestimate the region of the image frame corresponding to the person(including parts of the person that are occluded in the image frame)based on other frames of the video, and the trained machine learningmodel may be used to analyze the video and select the region of theimage frame corresponding to the person together with parts of theperson that are occluded in the image. An example of such trainingexample may include a video together with a desired selection of aregion in an image frame of the video corresponding to a personincluding parts of the person that are occluded in that image frame. Forexample, Step 1520 may present an image of a person behind a fruitstand, and Step 1530 may receive an indication of the person (forexample by an “x” marker pointing to the depiction of the person in theimage), as illustrated in FIG. 16D. Further, in this example, inresponse to an indication of a selected object type of a personincluding occluded parts received by Step 1510 and the input “x” markerreceived by Step 1530, a region of the image corresponding to the personincluding occluded parts of the person may be selected by Step 1540 asillustrated in FIG. 16D. Further, in this example, in response to anindication of a selected object type of a person except for occludedparts of the person received by Step 1510 and the input “x” markerreceived by Step 1530, a region of the image corresponding to the personwithout the occluded parts of the person may be selected by Step 1540 asillustrated in FIG. 16F.

In yet another example, the first selected object type of Step 1510 maybe male and the second selected object type of Step 1510 may be female,in response to the first selected object type and the input to theobject selection tool received by Step 1530, a first region of the imagecorresponding to a male figure in the image may be selected by Step1540, and in response to the second selected object type and the inputto the object selection tool received by Step 1530, a second region ofthe image corresponding to a female figure in the image may be selectedby Step 1540, and therefore the first region may differ from the secondregion.

In an additional example, the first selected object type of Step 1510may be face including hair and the second selected object type of Step1510 may be face except for hair, in response to the first selectedobject type and the input to the object selection tool received by Step1530, a first region of the image corresponding to a face and the haircorresponding to the face in the image may be selected by Step 1540, andin response to the second selected object type and the input to theobject selection tool received by Step 1530, a second region of theimage corresponding to the face except for the hair corresponding to theface in the image may be selected by Step 1540, and therefore the firstregion may differ from the second region.

In yet another example, the first selected object type of Step 1510 maybe a single instance of an object of a specified object type and thesecond selected object type of Step 1510 may be one or more instances ofobjects of the specified object type, in response to the first selectedobject type and the input to the object selection tool received by Step1530, a first region of the image corresponding to a single instance ofan object in the image may be selected by Step 1540, and in response tothe second selected object type and the input to the object selectiontool received by Step 1530, a second region of the image correspondingto a plurality of instances of the object in the image may be selectedby Step 1540, and therefore the first region may differ from the secondregion.

In an additional example, the first selected object type of Step 1510may be a person and the second selected object type of Step 1510 may bea vehicle, in response to the first selected object type and the inputto the object selection tool received by Step 1530, a first region ofthe image corresponding to a person in the image may be selected by Step1540, and in response to the second selected object type and the inputto the object selection tool received by Step 1530, a second region ofthe image corresponding to a vehicle in the image may be selected byStep 1540, and therefore the first region may differ from the secondregion.

In yet another example, the first selected object type of Step 1510 maybe a car and the second selected object type of Step 1510 may be atruck, in response to the first selected object type and the input tothe object selection tool received by Step 1530, a first region of theimage corresponding to a car in the image may be selected by Step 1540,and in response to the second selected object type and the input to theobject selection tool received by Step 1530, a second region of theimage corresponding to a truck in the image may be selected by Step1540, and therefore the first region may differ from the second region.

In some examples, the input to the object selection tool received byStep 1530 may comprise one or more points in the image (for example theinput may indicate that the one or more points are part of the object),and the region of the image selected by Step 1540 may include the one ormore points and at least one additional pixel of the image. Further, insome examples, the input to the object selection tool received by Step1530 may comprise one or more additional points in the image (forexample the input may indicate that the one or more additional pointsare not part of the object), and the region of the image selected byStep 1540 may not include any of the one or more additional points. Forexample, selecting the region of the image selected by Step 1540 maycomprise solving an optimization problem that optimize some criterionfunction over the selected region under some constraints about the validselected regions, and in some examples the constraints may specify thatthe one or more points must be within the selected region and/or thatthe one or more additional points must be outside the selected region.

In some examples, the input to the object selection tool received byStep 1530 may comprise one or more scribbles in the image (for examplethe input may indicate that the one or more areas indicated by thescribbles are part of the object), and the region of the image selectedby Step 1540 may include the one or more scribbles and at least oneadditional pixel of the image. Further, in some examples, the input tothe object selection tool received by Step 1530 may comprise one or moreadditional scribbles in the image (for example the input may indicatethat the one or more areas indicated by the additional scribbles are notpart of the object), and the region of the image selected by Step 1540may not include any of the one or more additional scribbles. Forexample, selecting the region of the image selected by Step 1540 maycomprise solving an optimization problem that optimize some criterionfunction over the selected region under some constraints about the validselected regions, and in some examples the constraints may specify thatthe one or more areas indicated by the scribbles must be within theselected region and/or that the one or more areas indicated by theadditional scribbles must be outside the selected region.

In some examples, the input to the object selection tool received byStep 1530 may comprise a bounding polygon (such as a bounding box, abounding triangle, etc.), and the region of the image selected by Step1540 may be included within the bounding polygon. For example, selectingthe region of the image selected by Step 1540 may comprise solving anoptimization problem that optimizes some criterion function over theselected region under some constraints about the valid selected regions,and in some examples the constraints may specify that the selectedregion must be contained within the bounding polygon.

In some embodiments, after the selection of the first region of theimage corresponding to the object of the first selected object type inthe image by Step 1540, an indication of a desire to change the selectedobject type to an updated object type may be received from the user(where the updated object type differs from the first selected objecttype), and in response to the received indication of the desire tochange the selected object type, the updated object type and the inputto the object selection tool may be used to select a modified region ofthe image corresponding to an object of the updated object type in theimage. For example, Step 1540 may be repeated with the updated objecttype and the input to the object selection tool to select the modifiedregion of the image corresponding to the object of the updated objecttype in the image.

In some examples, an overlay may be presented over the presentation ofthe image, and the overlay may indicate the region of the image selectedby Step 1540. For example, the overlay may include a bounding contour ofthe selected region. In another example, the selected region may bepresented with a first color scheme while the rest of the image may bepresented with a second color scheme. In yet another example, theselected region may be presented with a first blurring settings (forexample, without blurring, with low blurring, etc.) while the rest ofthe image may be presented with a second blurring settings. In anadditional example, an overlay may include a first pattern over theselected region (for example, no pattern, dots pattern, lines pattern,and so forth) and a different pattern over the rest of the image (forexample, no pattern, dots pattern, lines pattern, and so forth).

In some examples, a dataset of images and masks (for example of datasets610) may be updated according to the region of the image selected byStep 1540. For example, the image presented by Step 1520 and a maskcorresponding to the region of the image selected by Step 1540 may beadded to the dataset. In another example, the image presented by Step1520 may already be part of the dataset, and a mask corresponding to theregion of the image selected by Step 1540 may be added to the dataset inconnection to the image and/or may replace another mask in the datasetthat is connected with the image.

In some embodiments, the image presented by Step 1520 and the input tothe object selection tool received by Step 1530 may be analyzed todetermine whether a region of the image corresponding to the input tothe object selection tool depicts an object of the selected object typeof Step 1510. Further, in some examples, in response to a determinationthat the region of the image corresponding to the input to the objectselection tool does not depict an object of the selected object type, anindication may be provided to the user. For example, the indication maybe provided visually, may be provided audibly, may be providedtextually, may be provided through a personalized device associated witha user, may be provided through a public device, and so forth. In someexamples, a machine learning model may be trained using trainingexamples to determine whether regions of images indicated by inputs toobject selection tools depict objects of particular types, and thetrained machine learning model may be used to analyze the imagepresented by Step 1520 and the input to the object selection toolreceived by Step 1530 to determine whether the region of the imagecorresponding to the input to the object selection tool depicts anobject of the selected object type of Step 1510. An example of suchtraining example may include an image and an input to the objectselection tool together with an indication of whether the region of theimage corresponding to the input to the object selection tool depicts anobject of a selected object type. In another example, an artificialneural network may be configured to determine whether regions of imagesindicated by inputs to object selection tools depict objects ofparticular types, and the artificial neural network may be used toanalyze the image presented by Step 1520 and the input to the objectselection tool received by Step 1530 to determine whether the region ofthe image corresponding to the input to the object selection tooldepicts an object of the selected object type of Step 1510.

FIG. 17 illustrates an example of a method 1700 for selecting images formanual annotation. In this example, method 1700 may comprise: accessingat least four images (Step 1710); presenting at least two images of theat least four images (Step 1720); receiving annotation data for thepresented at least two images (Step 1730); analyzing the receivedannotation data (Step 1740); selecting additional images of the at leastfour images (Step 1750); presenting the selected additional images (Step1760); and receiving annotation data for the selected additional images(Step 1770). In some implementations, method 1700 may comprise one ormore additional steps, while some of the steps listed above may bemodified or excluded. For example, in some cases Step 1710 and/or Step1720 and/or Step 1750 and/or Step 1760 may be excluded from method 1700.In some implementations, one or more steps illustrated in FIG. 17 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, in Step 1710 a group of images may be accessed.Further, in some examples, points in time corresponding to the group ofimages may also be accessed. For example, the group of images maycomprise a first image that may correspond to a first point in time, asecond image that may correspond to a second point in time and at leasttwo more images, each image of the at least two more images maycorrespond to a point in time. For example, an image may correspond to atime of capturing of the image, may correspond to a time of processingof the image, may correspond to a time associated with an objectdepicted in the images (for example, a sample may be depicted in theimage and the image may correspond to the sampling time of the sample, adocument may be depicted in the image and the image may correspond toediting time of the document, etc.), and so forth. In another example,the images may be frames in video, and the images may correspond to timeindices of the frames within the video. For example, in Step 1710, thegroup of images and/or the points in time corresponding to the group ofimages may be read from memory (for example from memory unit 210, sharedmemory module 410, etc.), read from a blockchain, received through acommunication network (such as communication network 130) using acommunication device (such as communication module 230), received froman external device (such as mobile phone 111, tablet 112, PC 113, remotestorage 140, NAS 150, server 300, cloud platform 400, etc.), and soforth. In another example, in Step 1710 the group of images may becaptured using image sensors (such as image sensor 260), may beextracted from a video, and so forth. In yet another example, in Step1710 metadata associated with the images (such as file name, timeassociated with the image as described above, description of the image,and so forth) may be accessed.

In some embodiments, in Step 1720 at least part of at least some of theimages of the group of images accessed by Step 1710 may be presented toa user. For example, at least part of the first image of the group ofimages obtained by Step 1710 and/or at least part of the second image ofthe group of images accessed by Step 1710 may be presented to the user.For example, (the parts of) the images may be presented through a userinterface, through a web service, through a display screen, through anaugmented reality system, through a personalized device of the user,through a public device, through an external device, and so forth.

In some embodiments, in Step 1730 annotation data associated with theimages of the group of images accessed by Step 1710 and/or presented byStep 1720 may be received from the user. For example, first annotationdata associated with the first image of the group of images obtained byStep 1710 and/or second annotation data associated with the second imageof the group of images obtained by Step 1710 may be received from theuser. For example, in Step 1730 the annotation data may be read frommemory (for example from memory unit 210, shared memory module 410,etc.), read from a blockchain, received through a communication network(such as communication network 130) using a communication device (suchas communication module 230), received from an external device (such asmobile phone 111, tablet 112, PC 113, remote storage 140, NAS 150,server 300, cloud platform 400, etc.), and so forth. For example, theannotation data of Step 1730 and/or the annotation data of Step 1770 maycomprise a label of an image and/or of an item depicted in the image, atag of an image and/or of an item depicted in the image, aclassification of the image and/or of an item depicted in the image, anindication of a location of an object within the image (for example as apoint, a mask, a bounding polygon, and so forth), a value associatedwith the image and/or of an item depicted in the image, a textualdescription of the image and/or of an item depicted in the image, asegmentation of the image, and so forth.

In some embodiments, in Step 1740 the annotation data received by Step1730 and associated with the images of the group of images accessed byStep 1710 and/or presented by Step 1720 may be analyzed. In someexamples, in Step 1740 the first annotation data and the secondannotation data received by Step 1730 may be compared. For example, alabel of the first annotation data may be compared with a label of thesecond annotation data to determine whether the two are identical or notand/or to calculate a measure of similarity between the two, a tag ofthe first annotation data may be compared with a tag of the secondannotation data to determine whether the two are identical or not and/orto calculate a measure of similarity between the two, a classificationdata of the first annotation data may be compared with a classificationdata of the second annotation data to determine whether the two areidentical or not and/or to calculate a measure of similarity between thetwo, a value of the first annotation data may be compared with a valueof the second annotation data to determine whether the two are identicalor not and/or to calculate a measure of similarity between the two, alocation within a first image of the first annotation data may becompared with a location within a second image of the second annotationdata to determine whether the two are identical or not and/or tocalculate a measure of similarity between the two, a region of a firstimage of the first annotation data may be compared with a region of asecond image of the second annotation data to determine whether the twoare identical or not and/or to calculate a measure of similarity betweenthe two, textual information of the first annotation data may becompared with textual information of the second annotation data todetermine whether the two are identical or not and/or to calculate ameasure of similarity between the two, and so forth. For example, asimilarity between two values may be calculated as a monotonicallydecreasing function of a distance between the two values, as a functionof a correlation between the two values, and so forth. In anotherexample, a similarity between two regions may be calculated using theJaccard index of the two regions, using the Sørensen-Dice coefficient ofthe two regions, using the Tversky index of the two regions, and soforth. In yet another example, a similarity between two texts may becalculated using a most-frequent-k-characters distance, hammingdistance, distance between the two texts in an embedded space (forexample, embedded space created using word2vec algorithm), and so forth.

In some embodiments, in Step 1750 additional images may be selected ofthe group of images accessed by Step 1710. For example, a third imagemay be selected of the at least two more images accessed by Step 1710.For example, the additional images (such as the third image above) maybe selected based on, for example, a result of the analysis of Step 1740(for example, based on a result of the comparison of the firstannotation data and the second annotation data by Step 1740), and thepoints in time corresponding to the group of images accessed by Step1710 (for example, the first point in time, the second point in time,and the points in time corresponding to at least some of the at leasttwo more images). For example, a machine learning model may be trainedusing training examples to select images of a plurality of alternativeimages based on comparison results and/or points in time, and thetrained machine learning model may be used to analyze the comparison ofthe first annotation data and the second annotation data by Step 1740and/or and the points in time corresponding to the group of imagesaccessed by Step 1710 and select the additional images of the at leasttwo more images accessed by Step 1710. An example of such trainingexample may comprise comparison results and/or points in time, togetherwith a desired selection of images of a plurality of alternative images.In another example, an artificial neural network may be configured toselect images of a plurality of alternative images based on comparisonresults and/or points in time, and the artificial neural network may beused to analyze the comparison of the first annotation data and thesecond annotation data by Step 1740 and/or and the points in timecorresponding to the group of images accessed by Step 1710 and selectthe additional images of the at least two more images accessed by Step1710. In yet another example, the selection of the additional images ofthe at least two more images accessed by Step 1710 by Step 1750 may bebased on one or more of a comparison of a label of the first annotationdata with a label of the second annotation data as described above, acomparison of a tag of the first annotation data with a tag of thesecond annotation data as described above, a comparison of aclassification data of the first annotation data with a classificationdata of the second annotation data as described above, a comparison of avalue of the first annotation data with a value of the second annotationdata as described above, a comparison of a location within a first imageof the first annotation data with a location within a second image ofthe second annotation data as described above, a comparison of a regionwithin a first image of the first annotation data with a region within asecond image of the second annotation data as described above (forexample based on one or more of the Jaccard index of the two regions,the Sørensen-Dice coefficient of the two regions, the Tversky index ofthe two regions, etc.), a comparison of textual information of the firstannotation data with textual information of the second annotation dataas described above, and so forth.

In some examples, in Step 1750, in response to a first result of theanalysis of Step 1740 (for example, in response to a first result of thecomparison of the first annotation data and the second annotation databy Step 1740), the third image of the at least two more images may beselected such that the third image corresponds to a point in timebetween the first point in time and the second point in time, and inresponse to a second result of the analysis of Step 1740 (for example,in response to a second result of the comparison of the first annotationdata and the second annotation data), the third image of the at leasttwo more images corresponding to a third point in time may be selectedsuch that the second point in time is between the first point in timeand the third point in time.

In some examples, in Step 1750, in response to a first result of theanalysis of Step 1740 (for example, in response to a first result of thecomparison of the first annotation data and the second annotation databy Step 1740), the third image of the at least two more images may beselected such that the third image corresponds to a point in time closerto the first point in time than to the second point in time, and inresponse to a second result of the analysis of Step 1740 (for example,in response to a second result of the comparison of the first annotationdata and the second annotation data), the third image of the at leasttwo more images may be selected such that the third image corresponds toa point in time closer to the second point in time than to the firstpoint in time.

In some embodiments, in Step 1760 at least part of the additional imagesselected by Step 1750 may be presented to the user. For example, in Step1760 at least part of the third image selected by Step 1750 may bepresented to the user. For example, at least part of the additionalimages selected by Step 1750 may be presented to the user. For example,(the parts of) the additional images may be presented through a userinterface, through a web service, through a display screen, through anaugmented reality system, through a personalized device of the user,through a public device, through an external device, and so forth.

In some embodiments, in Step 1770 annotation data associated with theadditional images selected by Step 1750 and/or presented by Step 1760may be received from the user. For example, third annotation dataassociated with the third image selected by Step 1750 and/or presentedby Step 1760 may be received from the user. For example, in Step 1770the annotation data may be read from memory (for example from memoryunit 210, shared memory module 410, etc.), read from a blockchain,received through a communication network (such as communication network130) using a communication device (such as communication module 230),received from an external device (such as mobile phone 111, tablet 112,PC 113, remote storage 140, NAS 150, server 300, cloud platform 400,etc.), and so forth.

In some examples, a dataset of images and annotations (for example ofdatasets 610) may be updated according to the additional images selectedby Step 1750 and/or presented by Step 1760 (such as the selected thirdimage) and the annotation data associated with the additional images andreceived from the user by Step 1770 (such as the third annotation datareceived from the user by Step 1770). For example, the image and theannotation data may be added to the dataset. In another example, theimage may already be part of the dataset, and the annotation data may beadded to the dataset in connection to the image and/or may replaceanother annotation data in the dataset that is connected with the image.

In some embodiments, a quantity may be selected based on a result of theanalysis of Step 1740 (for example, based on a result of the comparisonof the first annotation data and the second annotation data by Step1740), for example as described below. Further, in some examples, inStep 1750 a set of images may be selected of the at least two moreimages accessed by Step 1710, such that a quantity of images in theselected set of images matches the selected quantity, for example asdescribed below. Further, in some examples, the selected set of imagesmay be presented to the user, for example as described above in relationto Step 1720. Further, in some examples, annotation data associated witheach of the selected set of images may be received from the user, forexample as described above in relation to Step 1730.

In some examples, a quantity may be selected based on a result of theanalysis of Step 1740 (for example, based on a result of the comparisonof the first annotation data and the second annotation data by Step1740). For example, in response to a first result of the analysis ofStep 1740 (for example, a first result of the comparison of the firstannotation data and the second annotation data by Step 1740), a firstquantity may be selected, and in response to a second result of theanalysis of Step 1740 (for example, a second result of the comparison ofthe first annotation data and the second annotation data by Step 1740),a second quantity may be selected. In another example, the quantity maybe a function of the result of the analysis of Step 1740 (for example, afunction of a result of the comparison of the first annotation data andthe second annotation data by Step 1740). In yet another example, amachine learning model may be trained to select quantities based oncomparison results, and the trained machine learning model may be usedto analyze the comparison result of Step 1740 and select the quantity.

In some examples, in Step 1750 a set of images may be selected of the atleast two more images accessed by Step 1710, such that a quantity ofimages in the selected set of images matches the selected quantity. Forexample, the selection of the set of images may be performed by Step1750 as described above. In another example, Step 1750 may select aquantity of images that is higher than the desired quantity, the imagesmay be ranked, and the images with the highest rank may be picked suchthat the quantity of picked images matches the selected quantity. Forexample, Step 1750 may use a machine learning model as described above,many machine learning models may further produce a confidence level foreach selected image, and such confidence levels may be used for rankingthe images (having a higher rank for an image corresponding to a higherconfidence). In yet another example, the selected (parts of) images maybe ranked according to size, assigning a higher rank for larger images.

In some embodiments, an embedding in a mathematical space of the groupof images accessed by Step 1710 may be determined, for example asdescribed below. Further, in some examples, in Step 1750 the determinedembedding of the images in the mathematical space may be used to selectadditional images of the group of images accessed by Step 1710, forexample as described below.

In some examples, an embedding in a mathematical space of the group ofimages accessed by Step 1710 may be determined. For example, anembedding in a mathematical space of at least the first image, thesecond image and the at least two more images accessed by Step 1710 maybe determined, for example as described below.

In some examples, the embedding in the mathematical space of the groupof images accessed by Step 1710 may be determined based on pixel data ofthe images in the group of images. For example, the embedding in themathematical space of at least the first image, the second image and theat least two more images accessed by Step 1710 may be determined basedon pixel data of the first image, pixel data of the second image andpixel data of at least some of the at least two more images.

In some examples, the embedding in the mathematical space of the groupof images accessed by Step 1710 may be determined based on pixel data ofthe images in the group of images and points in time corresponding tothe group of images accessed by Step 1710. For example, the embedding inthe mathematical space of at least the first image, the second image andthe at least two more images accessed by Step 1710 may be determinedbased on pixel data of the first image, pixel data of the second image,pixel data of at least some of the at least two more images, the firstpoint in time, the second point in time, and the points in timecorresponding to at least some of the at least two more images. In someexamples, a mathematical vector corresponding to an image may beconcatenated with a mathematical value and/or a mathematical vectorcorresponding to a point in time, and the resulting mathematical vectorfor the relevant images and points in time (for example as describedabove) may be used to determine the embedding in the mathematical spaceof the group of images accessed by Step 1710, for example usingprincipal component analysis (PCA), using Fisher's (linear discriminantanalysis) LDA, using minor component analysis (MCA), using t-distributedstochastic neighbor embedding (t-SNE), and so forth. For example, themathematical vector corresponding to an image may be generated by amathematical vectorization of a matrix, where each entry in the matrixcorresponds to a pixel in the image and comprise a value related to thatpixel (such as gray level of the pixel, value of a color component ofthe pixel, value of an edge detector for that pixel, value of aconvolution for that pixel, and so forth). In another example, themathematical vector corresponding to an image may be generated by anobject detector and comprise information about objects detected in theimage by the object detector, such as number of detected objects,locations of detected objects, type of detected objects, properties ofdetected objects (such as size, color, etc.), and so forth. In yetanother example, the mathematical vector corresponding to a point intime may comprise a single value corresponding to the number of timeunits passed since a particular time to the point in time, a vector withvalues describing the point in time (such as second, minute, hour, day,month, year, and so forth).

In some examples, in Step 1750 the determined embedding of the images inthe mathematical space may be used to select additional images of the atleast two more images accessed by Step 1710. For example, in Step 1750the determined embedding in the mathematical space of at least the firstimage, the second image and the at least two more images may be used toselect the third image of the at least two more images. In someexamples, a centroid in the mathematical space of the embedding of thefirst image, the embedding of the second image, and in some examplesembedding of other images, may be calculated. Further, the centroid maybe used by Step 1750 to select additional images of the at least twomore images accessed by Step 1710, for example by selecting the imagesclosest to the centroid of the at least two more images accessed by Step1710, by selecting the images farthest from the centroid, and so forth.In some examples, the images of at least two more images accessed byStep 1710 with the maximal minimal distance to the any one of the firstimage accessed by Step 1710 and the second image accessed by Step 1710.

In some embodiments, the group of images accessed by Step 1710 may be agroup of frames of a video. For example, the first image may be a firstframe of the video, the second image may be a second frame of the video,and the at least two more images may be at least two more frames of thevideo. Further, in some examples, in response to a first result of theanalysis of Step 1740 (for example, in response to a first result of thecomparison of the first annotation data and the second annotation data),a first frame rate may be selected, and in response to a second resultof the analysis of Step 1740 (for example, in response to a secondresult of the comparison of the first annotation data and the secondannotation data), a second frame rate may be selected, where the secondframe rate may differ from the first frame rate. Further, in someexamples, frames of at least part of the video may be sampled at theselected frame rate, for example as described below. Further, in someexamples, the sampled frames may be presented to the user. For example,the obtained video of the selected frame rate may be played using avideo player. In another example, individual frames of the sampledframes may be presented to the used, for example as described inrelation to Step 1760. Further, in some examples, annotation dataassociated with the sampled frames may be received from the user. Forexample, annotation data related to the entire obtained video may bereceived (for example, with annotation data related to an actiondepicted in the obtained video, related to an event depicted in theobtained video, to motion depicted in the obtained video, and so forth).In another example, annotation data related to individual frames of thesampled frames may be received, for example as described in relation toStep 1770.

In some examples, frames of at least part of the video may be sampled atthe selected frame rate. For example, the at least part of the video maybe in a frame rate that is higher than the selected frame rate, and someof the frames of the at least part of the video may be dropped to obtaina video of the selected frame rate. In another example, the at leastpart of the video may be in a frame rate that is lower than the selectedframe rate, intra-frames interpolation may be used to generateadditional frames, and the generated additional frames may be added tothe at least part of the video to obtain a video of the selected framerate.

In some embodiments, in response to a first result of the analysis ofStep 1740 (for example, in response to a first result of the comparisonof the first annotation data and the second annotation data), a firstimage resolution may be selected, and in response to a second result ofthe analysis of Step 1740 (for example, in response to a second resultof the comparison of the first annotation data and the second annotationdata), a second image resolution may be selected, where the second imageresolution may differ from the first image resolution. Further, in someexamples, at least part of the selected third image may be presented inthe selected resolution by Step 1760. Further, third annotation dataassociated with the third image may be received from the user asdescribed above in relation to Step 1770, for example in response to thepresentation of the at least part of the selected third image in theselected resolution. For example, the third image may be in an imageresolution that is higher than the selected image resolution, and someof the pixels of the third image may be dropped to obtain an image ofthe selected image resolution. In another example, the third image maybe in an image resolution that is lower than the selected imageresolution, pixel interpolation may be used to generate additionalpixels, and the generated additional pixels may be added to the thirdimage to obtain an image of the selected image resolution.

In some embodiments, in response to a first result of the analysis ofStep 1740 (for example, in response to a first result of the comparisonof the first annotation data and the second annotation data), a firstimage augmentation may be selected, and in response to a second resultof the analysis of Step 1740 (for example, in response to a secondresult of the comparison of the first annotation data and the secondannotation data), a second image augmentation may be selected, where thesecond image augmentation may differ from the first image augmentation.Further, in some examples, the selected image augmentation may beapplied to at least part of the third image to obtain an augmentedimage. Further, in some examples, at least part of the obtainedaugmented image may be presented to the user by Step 1760. Further, insome examples, third annotation data associated with the third image maybe received from the user as described above in relation to Step 1770,for example in response to the presentation of the at least part of theobtained augmented image. Some examples of such image augmentations mayinclude shifting the image, flipping the image, rotating the image,zooming, changing brightness, blurring the image, sharpening the image,applying a convolution to the image, cropping the image, and so forth.

In some embodiments, the first image and the second image accessed byStep 1710 and the first annotation data and the second annotation datareceived by Step 1730 may be used to train a machine learning algorithmand obtain an inference model, for example as described above with thefirst image and the second image with the corresponding annotations usedas training examples in the training of the machine learning algorithm.Further, in some examples, the obtained inference model may be used toobtain at least one inferred result for each image of the at least twomore images accessed by Step 1710, for example as described above.Further, in some examples, in Step 1750 the third image of the at leasttwo more images may be selected based on a result of the comparison ofthe first annotation data and the second annotation data, the firstpoint in time, the second point in time, the points in timecorresponding to at least some of the at least two more images and theobtained inferred results associated with at least some of the at leasttwo more images. For example, a machine learning model may be trainedusing training examples to select images of a plurality of alternativeimages based on comparison results and/or points in time and/or inferredresults associated with the alternative images, and the trained machinelearning model may be used to analyze the comparison of the firstannotation data and the second annotation data by Step 1740 and/or andthe points in time corresponding to the group of images accessed by Step1710 and/or inferred results associated with at least some of the atleast two more images and select the additional images of the at leasttwo more images accessed by Step 1710. An example of such trainingexample may comprise comparison results and/or points in time and/orinferred results, together with a desired selection of images of aplurality of alternative images. In another example, an artificialneural network may be configured to select images of a plurality ofalternative images based on comparison results and/or points in timeand/or inferred results associated with the alternative images, and theartificial neural network may be used to analyze the comparison of thefirst annotation data and the second annotation data by Step 1740 and/orand the points in time corresponding to the group of images accessed byStep 1710 and/or inferred results associated with at least some of theat least two more images and select the additional images of the atleast two more images accessed by Step 1710.

In some embodiments, the first image and the second image accessed byStep 1710 and the first annotation data and the second annotation datareceived by Step 1730 may be used to train a machine learning algorithmand obtain an inference model, for example as described above. Further,in some examples, at least two augmented images may be generated foreach image of the at least two more images accessed by Step 1710, forexample as described above. Further, in some examples, the obtainedinference model may be used to obtain a plurality of inferred resultsfor each image of the at least two more images accessed by Step 1710,wherein the plurality of inferred results corresponding to anyparticular image of the at least two more images may comprise at leastone inferred result for each augmented image of the generated at leasttwo augmented images corresponding to the particular image. For example,each augmented image of the generated at least two augmented imagescorresponding to the particular image may be used as input to theobtained inference model to obtain an inferred result, for example asdescribed above. Further, in some examples, the plurality of inferredresults of each image of the at least two more images accessed by Step1710 may be analyzed to obtain a stability assessment for each image ofthe at least two more images, for example as described below. Further,in some examples, in Step 1750 the stability assessments associated withat least some of the at least two more images accessed by Step 1710 maybe used to select the third image of the at least two more images.

In some examples, the plurality of inferred results of each image of theat least two more images accessed by Step 1710 may be analyzed to obtaina stability assessment for each image of the at least two more images.For example, a plurality of inferred results may comprise two inferredresults, the two inferred results may be compared to determine a measureof similarity (or a measure of dissimilarity) between the two inferredresults, and the stability assessment may be based on the determinedmeasure of similarity (or the determined measure of dissimilarity), forexample by taking a function of the measure as the stability assessment.In another example, a statistical measure may be calculated for aplurality of inferred results (such as standard deviation, variance, andso forth), and the stability assessment may be based on the calculatedstatistical measure, for example by taking a function of the statisticalmeasure as the stability assessment.

In some examples, in Step 1750 the stability assessments associated withat least some of the at least two more images accessed by Step 1710 maybe used to select the third image of the at least two more images. Forexample, the image with the least stable plurality of inferred resultsof the at least two more images accessed by Step 1710 according to thestability assessments may be selected by Step 1750 as the third image.In another example, the stability according to the stability assessmentof a plurality of inferred results corresponding to an image may becombined with additional information corresponding to the image (forexample, information as described above) to obtain a combined score forthe image, and Step 1750 may select the third image of the at least twomore images based on the combined score (for example, selecting theimage with the highest combined score, the image with the lowestcombined score, the image with the combined score closest to aparticular value, the image with the combined score closest to the meanof the combined scores, the image with the combined score closest to themedian of the combined scores, and so forth).

What is claimed is:
 1. A method for selective update of inferencemodels, the method comprising: obtaining a first inference model and asecond inference model, the first inference model and the secondinference model are based on a first version of a set of trainingexamples; obtaining an updated version of the set of training examples;based on the updated version of the set of training examples, generatingan updated version of the first inference model; evaluating the updatedversion of the first inference model to obtain an evaluation result; andusing the evaluation result to determine whether to update the secondinference model.
 2. The method of claim 1, wherein the first inferencemodel is a result of training a machine learning algorithm using a firsttraining set, the second inference model is a result of training themachine learning algorithm using a second training set, the secondtraining set differs from the first training set, and both the firsttraining set and the second training set comprises the set of trainingexamples.
 3. The method of claim 1, wherein the first inference model isa result of training a first machine learning algorithm using the set oftraining examples, the second inference model is a result of training asecond machine learning algorithm using the set of training examples,the updated version of the first inference model is a result of trainingthe first machine learning algorithm using the updated version of theset of training examples, and wherein updating the second inferencemodel comprises training the second machine learning algorithm using theupdated version of the set of training examples.
 4. The method of claim1, wherein the first inference model is a result of training a machinelearning algorithm using the set of training examples and a firsthyper-parameter of the machine learning algorithm, the second inferencemodel is a result of training the machine learning algorithm using theset of training examples and a second hyper-parameter of the machinelearning algorithm, the updated version of the first inference model isa result of training the machine learning algorithm using the updatedversion of the set of training examples and the first hyper-parameter ofthe machine learning algorithm, and wherein updating the secondinference model comprises training the machine learning algorithm usingthe updated version of the set of training examples and the secondhyper-parameter of the machine learning algorithm.
 5. The method ofclaim 1, wherein the first inference model is used by a firstpersonalized device, the second inference model is used by a secondpersonalized device, and further comprising: in response to a firstevaluation result, causing the second personalized device to use anupdated version of the second inference model; and in response to asecond evaluation result, forgoing causing the second personalizeddevice to use the updated version of the second inference model.
 6. Themethod of claim 1, wherein the first inference model is used by a firstpersonalized device, the second inference model is used by a secondpersonalized device, and further comprising: causing the firstpersonalized device to use the updated version of the first inferencemodel; receiving results of the usage of the updated version of thefirst inference model by the first personalized device; and using thereceived results of the usage of the updated version of the firstinference model by the first personalized device to evaluate the updatedversion of the first inference model and obtain the evaluation result.7. The method of claim 1, wherein the first inference model and thesecond inference model are used by a first personalized device, andfurther comprising causing a second personalized device to use theupdated version of the first inference model and an updated version ofthe second inference model.
 8. The method of claim 1, furthercomprising: using the first inference model to analyze a plurality ofinputs and produce a first plurality of outputs corresponding to theinputs; using the updated version of the first inference model toanalyze the plurality of inputs and produce a second plurality ofoutputs corresponding to the inputs; and using the first plurality ofoutputs produced using the first inference model and the secondplurality of outputs produced using the updated version of the firstinference model to evaluate the updated version of the first inferencemodel and obtain the evaluation result.
 9. The method of claim 1,further comprising: using the updated version of the first inferencemodel to analyze a plurality of inputs and produce a plurality ofoutputs corresponding to the inputs; providing information based on theplurality of outputs to at least one user; receiving one or morefeedbacks related to the information based on the plurality of outputsfrom the at least one user; and using the one or more feedbacks relatedto the information based on the plurality of outputs to evaluate theupdated version of the first inference model and obtain the evaluationresult.
 10. The method of claim 1, wherein the first inference model isconfigured to analyze visual data captured from an environment, and thesecond inference model is configured to analyze audio data captured inconjunction with the visual data.
 11. The method of claim 1, wherein thefirst inference model is configured to analyze audio data captured froman environment, and the second inference model is configured to analyzevisual data captured in conjunction with the audio data.
 12. The methodof claim 1, wherein the first inference model is configured to analyzeaudio data captured from an environment, and the second inference modelis configured to analyze textual information associated with the audiodata.
 13. The method of claim 1, wherein the first inference model isconfigured to analyze image data captured from an environment, and thesecond inference model is configured to analyze textual informationassociated with the image data.
 14. The method of claim 1, wherein thefirst inference model is configured to detect a first type of items inimage data captured from an environment, and the second inference modelis configured to determine properties of the detected items by analyzingthe image data.
 15. The method of claim 1, wherein the first version ofthe set of training examples includes at least one example not includedin the updated version of the set of training examples, and furthercomprising using the at least one example not included in the updatedversion of the set of training examples to generate the updated versionof the first inference model.
 16. The method of claim 1, wherein theupdated version of the set of training examples includes at least oneexample not included in the first version of the set of trainingexamples, and further comprising using the at least one example notincluded in the first version of the set of training examples togenerate the updated version of the first inference model.
 17. A systemfor selective update of inference models, the system comprising: atleast one processor configured to: obtain a first inference model and asecond inference model, the first inference model and the secondinference model are based on a first version of a set of trainingexamples; obtain an updated version of the set of training examples;based on the updated version of the set of training examples, generatean updated version of the first inference model; evaluate the updatedversion of the first inference model to obtain an evaluation result; anduse the evaluation result to determine whether to update the secondinference model.
 18. The system of claim 17, wherein the first inferencemodel is configured to analyze visual data captured from an environment,and the second inference model is configured to analyze audio datacaptured in conjunction with the visual data.
 19. The system of claim17, wherein the first inference model is configured to analyze audiodata captured from an environment, and the second inference model isconfigured to analyze visual data captured in conjunction with the audiodata.
 20. A non-transitory computer readable medium storing data andcomputer implementable instructions for carrying out a method forselective update of inference models, the method comprising: obtaining afirst inference model and a second inference model, the first inferencemodel and the second inference model are based on a first version of aset of training examples; obtaining an updated version of the set oftraining examples; based on the updated version of the set of trainingexamples, generating an updated version of the first inference model;evaluating the updated version of the first inference model to obtain anevaluation result; and using the evaluation result to determine whetherto update the second inference model.